Information converting system

ABSTRACT

An information converting system includes: a database ( 300 ) in which attribute data including three-dimensional shape data of parts modeled on various objects and identification codes are registered; a comparison part generating unit ( 400 ) for generating part information for comparison from the attribute data for each of the parts; an input unit ( 100 ) for obtaining an input image including an object image; a comparison information generating unit ( 200 ) for performing an imaging process on the input image to thereby generate image information for comparison in which information of the object image is not clipped; and a specifying unit ( 500 ) for retrieving a part corresponding to the part information for comparison from the image information for comparison, recognizing the corresponding portion in the image information for comparison as an object image, and specifying a part having the part information for comparison as a part corresponding to the object image.

TECHNICAL FIELD

The present invention relates to an information converting system forautomatically recognizing an object on the basis of input informationhaving a physical quantity indicative of the properties of the object.More particularly, the invention relates to an information convertingsystem suitable for automatically recognizing an object havingthree-dimensional shape on the basis of an input image and, further,relates to a basic technique for an information transferring techniqueor an artificial intelligence by using the results of recognition of anobject.

BACKGROUND ART

Conventionally, in a monitoring system such as a security system,various sensors such as a monitor camera and an infrared sensor areused. By using the monitor camera and sensor, the presence or absence ofan intruder in a building or the like can be easily monitored ordetected from a remote place.

In recent years, by digitization of an image, image processingtechniques have advanced dramatically. As a result, a specific portionin an image can be enhanced or clipped, and synthesis of desired imageshas been made possible. For example, in live coverage of a baseballgame, a technique of arbitrarily replacing an advertisement image behindthe batter's box and broadcasting the resultant images are a practicaluse.

Further, because of the progress in communication techniques of recentyears, the amount of information transferred via a communication linesuch as the Internet is increasing. Particularly, the amount of imageinformation is incomparably larger than that of character information.Therefore, in order to reduce the amount of image informationtransmitted, various image compressing techniques for compressing animage signal, transmitting the compressed image signal, anddecompressing the image signal on the reception side have beendeveloped.

For example, as a compression encoding system for a still image, theJPEG (Joint Photographic coding Experts Group) system is adopted as aninternational standard system. In the JPEG system, the total amount ofimage information is reduced by thinning out the number of pixels inaccordance with a predetermined rule. Also as a compression encodingsystem for a moving image, for example, the MPEG (Motion Picture codingExperts Group) system is adopted as an international standard system. Inthe MPEG system, only the parts of an image that are in motion areprocessed, thereby reducing the total amount of image information.

Incidentally, for recognizing the occurrence of an accident or a crime,it is still necessary to watch a monitor image of a monitor camera by ahuman being. That is, the occurrence of an accident or the like is notrecognized by the monitor camera or the monitor image itself. Therefore,even if a monitor camera is installed, if the person monitoring thecamera is not watching the monitor image, the occurrence of an accidentor the like will be missed.

Also, although a security sensor such as an infrared sensor can detectintrusion of something, it is difficult to recognize “what” has beendetected. Because of this, security sensors often give out false alarms.That is, the security sensor detects not only an intruder but alsointrusion of an animal such as a dog.

In the final analysis, the cause of these problems is that “what objectis” is not being recognized automatically.

Furthermore, in order to enhance or clip a specific portion of a digitalimage by image processing, the operator has to designate the specificportion. Also, however a digital image is processed by image processing,the image itself is merely a set of pixel signals. Consequently, “what”the object is in an image is still recognized by a human being in amanner similar to the case of the above-described monitor camera.

Incidentally, as an image recognizing technique, the optical characterreader (OCR) has been practically used. Objects for recognition in theOCR are usually characters on a plain white sheet of paper. The OCRautomatically recognizes characters by using a pattern matching methodof comparing a character pattern clipped from an input image with areference pattern.

However, in the case of recognizing the image of an object existing inthree-dimensional space, the background of the object is not limited toplain white but is often a succession of lines from the outlines ofneighboring objects. In this case, it is often difficult to clip anindividual object image from the background. Therefore, even by directlyapplying a conventional pattern matching technique such as the OCR, itis not easy to recognize a three-dimensional object.

Also in conventional image compressing techniques, because processing isintended to compress image signals, the volume of compressed imageinformation transmitted is much larger than that of characterinformation. As a result, there are still problems such that it takesmuch time to transfer image information and that the burden on thetransmission line becomes heavy.

Incidentally, by the existing image recognizing techniques, it isimpossible to realize the function of recognizing a three-dimensionalobject from two-dimensional image information of that three-dimensionalobject, reading a large amount of the three-dimensional information ofthe object from the two-dimensional image information, and inferring thethree-dimensional object from the read information like a human being.That is, although current two-dimensional image recognizing techniquesare fairly advanced, using existing techniques, recognition is onlypossible to realize to such an extent that the name and kind of theobject can be recognized. It is difficult to recognize the object so asto be separated from the other objects and make three-dimensionalmeasurement of a physical quantity of the object and so on like a humanbeing does.

Therefore, if three-dimensional recognition in a real meaning includingnot only recognition to the extent of name and kind of an object butalso recognition of various attributes, three-dimensional shape, andposition of three-dimensional coordinates of an object is realized, bycombining the recognition with the current computer technology, anartificial intelligence technique of selecting a target object from aplurality of existing objects, recognizing the object, measuring theobject, and further, deriving one final conclusion from the positionalrelation and the meaning relation of the objects like a human being doesdaily can be realized.

DISCLOSURE OF THE INVENTION

The present invention has been achieved in consideration of the abovecircumstances and its first object is to provide an informationconverting system capable of automatically recognizing athree-dimensional object.

By the invention, not only a three-dimensional object is automaticallyrecognized but, further, it becomes possible to recognize an object soas to be distinguished from another object, determine athree-dimensional shape of even an unseen portion of the object,determine position coordinates of the object, reconstruct a plurality ofobjects into a plurality of corresponding three-dimensional CG (ComputerGraphics) including positions, and represent the resultant at a freeviewpoint.

A second object of the invention is to provide a technique capable oflargely reducing the amount of image information by informationtransformation and further realizing high-speed transfer of the imageinformation.

By combining the first and second objects, a technique realizingartificial intelligence for interpreting the situation of an object anddetermining a conclusion in place of a human being in a remote place orthe like can be provided.

After various investigations to achieve the first object, the inventorherein paid attention to a fact that when the conventional patternmatching method is applied to recognize a three-dimensional object, theprocess of individually clipping an object image from an input imagebecomes very difficult. The inventor herein therefore has sought for atechnique capable of automatically recognizing an object withoutclipping an object image from an input image and has achieved thepresent invention.

An information converting system disclosed in the first aspect of theinvention includes: a database in which attribute data including dataindicative of properties of an object and an identification code of eachof parts modeled on various objects are registered; a comparison partgenerating unit for generating one piece or a plurality of pieces ofpart information for comparison from the attribute data for each of theparts; an input unit for obtaining input information includinginformation regarding an object; a comparison information generatingunit for generating information for comparison in which information ofthe objects is not individually separated from each other, from theinput information; a part specifying unit for specifying a partcorresponding to the object by using the part information for comparisonand the information for comparison each having the same kind of data;and an output unit for outputting, as a result of recognition of theobject, the identification code and at least a part of the attributedata of the specified data, and the part specifying unit has: aretrieving unit for retrieving a corresponding portion which correspondsto at least a part of the part information for comparison from theinformation for comparison sequentially with respect to one or pluralpiece(s) of part information for comparison of one or plural part(s); arecognizing unit for recognizing, as an object, the correspondingportion in the information for comparison; and a specifying unit forspecifying a part having the part information for comparison as a partcorresponding to the object.

Thus, for example, in the case where an object is sound information,even when it is difficult to extract: the sound information as a targetfrom input information due to noise or the like, the correspondingportion in the input information is retrieved from the part side, sothat the sound information as a target can be specified.

Moreover, not only simply recognizing the name of the object, afterspecifying the part, a process can be performed by replacing inputinformation with the specified part. As a result, even data which is notincluded in input information, if it is included in attribute datapreliminarily given to the specified part, the data can be output.According to the invention, therefore, realization of more advancedobject recognition, for example image recognition, image understanding,further, sound recognition, sound understanding, and automatictranslation, can be expected. The invention can be expected to beapplied as the basic technology of the information processing techniquesuch as artificial intelligence to various fields.

(Image)

The information converting system disclosed in the second aspect of theinvention has: a database in which attribute data includingthree-dimensional shape data and an identification code of each of partsmodeled on various objects are registered; a comparison part generatingunit for generating one or plural piece(s) of part information forcomparison from the attribute data for each of the parts; an input unitfor obtaining an input image including an object image; a comparisonimage generating unit for generating image information for comparison inwhich information pieces of the objects are not individually clipped, byperforming an imaging process on the input image; a part specifying unitfor specifying a part corresponding to the object image by using thepart information for comparison and the image information for comparisoneach having the same kind of data; and an output unit for outputting, asa result of recognition of the object image, the identification code andat least a part of the attribute data of the specified part, and thepart specifying unit includes: a retrieving unit for retrieving acorresponding portion which corresponds to at least a part of the partinformation for comparison from the image information for comparisonsequentially with respect to one or plural piece(s) of part informationfor comparison of one or plural part(s); a recognizing unit forrecognizing, as an object image, the corresponding portion in the imageinformation for comparison; and a specifying unit for specifying a parthaving the part information for comparison as a part corresponding tothe object image.

As described above, in the information converting system of theinvention, the object image is recognized by converting the object datato a modeled part having attribute data. At that time, the imageinformation for comparison is retrieved by the part information forcomparison. A corresponding portion in the comparison image isrecognized as an object image and a corresponding part is specified.

Consequently, the object can be automatically recognized withoutclipping the object images from the input image. Therefore, even in thecase where it is difficult to individually clip an object image from aninput image, a three-dimensional object can be automatically recognized.

In the invention, the object is not limited to an actually existing one.For example, an image in virtual reality or the like can be used as aninput image. For example, as image information for comparison, it isalso possible to generate integral transform data from an input image bya method such as Fourier transform and generate, as part information forcomparison, integral transform data from attribute data of a part by amethod such as Fourier transform.

Further, according to the invention, attribute data is given to eachpart. Consequently, not only simply recognizing the name of the object,after specifying the part, a process can be performed by replacing inputimage with the specified part. As a result, even data which is notincluded in input image, if it is included in attribute datapreliminarily given to the specified part, the data can be output. Forexample, the shape of a backside portion of an object or information ofprice or weight of the object, which does not appear in the input image,can be also output as attribute data.

As described above, according to the invention, more advanced imagerecognition and image understanding can be realized. The invention canbe expected to be applied as the basic technique of the informationprocessing technique such as artificial intelligence to various fields.

The object is included in the object in the first aspect. The comparisonimage generating unit is included in the comparison informationgenerating unit in the first aspect. Also the image information forcomparison is included in the information for comparison in the firstaspect.

(Decomposition of Part Element)

According to the invention of the third aspect, the comparison partgenerating unit decomposes, as the part information for comparison, theattribute data of the part into basic elements of an outline or the likeand generates basic elements or a composite element obtained bycombining a plurality of basic elements. The comparison image generatingunit extracts the basic elements of an outline or the like and generatesa set of basic elements or composite elements as the image informationfor comparison. The retrieving unit retrieves a part corresponding tothe basic element or composite element of the part from the imageinformation for comparison.

In such a manner, when the attribute data is decomposed to the basiselements, and a corresponding portion in the comparison image isretrieved on the unit basis of the basic element or composite element,efficiency of the retrieving process can be improved. It is preferableto give an element identification code to each of the basic andcomposite elements for the following processes.

(Characteristic Element of Part)

According to the invention of the fourth aspect, the comparison partgenerating unit generates, as the part information for comparison, basicelements or a composite element of a characteristic portion of theattribute data of a part, the retrieving unit retrieves a partcorresponding to the basic element or composite element of thecharacteristic portion from the image information for comparison, andthe recognizing unit detects, after the portion corresponding to thebasic element or composite element of the characteristic portion isretrieved, correspondence between the corresponding portion and a basicelement or composite element out of the characteristic portion in thesame part, and recognizes the corresponding portion as an object image.

Consequently, by performing the retrieving process by using the basicelement or composite element in the characteristic portion, theefficiency of the retrieving process can be further increased.

(Part Operator)

According to the invention of the fifth aspect, the comparison partgenerating unit generates, as the part information for comparison, anelement extracting filter (hereinbelow, also called “element operator”)taking the form of a two-dimensional matrix or a three-dimensionalmatrix in which a high point is given to a pixel coinciding with theshape of the basic element or composite element and a low point is givento a pixel apart from the shape of the element, and the retrieving unitretrieves, as the corresponding portion, a portion in which the totalpoint of pixels coinciding with the basic element or composite elementin the image information for comparison is the highest.

By using the element extracting filter, while retrieving the portion inwhich the total point is the highest, coincidence can be finally made.Thus, with suppressing useless retrieval, the retrieving efficiency canbe improved.

(Coupling Relation)

According to the invention of the sixth aspect, the comparison partgenerating unit gives information for specifying only a couplingrelation of the basic elements to the composite element, and the partspecifying unit retrieves the corresponding portion on the conditionthat at least a part of the coupling relation coincides with thecorresponding portion.

In such a manner, when only the coupling relation is specified, theinformation of direction, size, position, and shape of a compositeelement can be ignored. Consequently, even in the case where thedirections, sizes, positions, shapes, or the like do not coincide, acorresponding portion which partially coincides with a composite elementcan be retrieved. As a result, the corresponding portion can beretrieved by the part information for comparison of a smaller number ofkinds. Thus, the efficiency of the retrieving process can be furtherincreased.

Further, when the element identification code is given to each of thebasic elements constructing a composite element and the elementidentification code is also given to each of the basic elements of theimage information for comparison obtained by decomposing the input imageinto the basic elements, coincidence is derived by each of the elementidentification codes. For example, the element identification code ofthe part information for comparison and the element identification codeof image information for comparison can be compared with each other on atable.

The basic element includes corner, line segment, plane or a combinationof these, or data obtained by integrally transforming them by a methodof the Fourier transform.

(Self-Recognizing Function)

According to the invention of the seventh aspect, the attribute data ofeach part registered in the database includes self-specifyinginformation for instructing a method of specifying the part, thecomparison part generating unit generates part information forcomparison for designating the self-specifying information and outputsthe part information for comparison to the part specifying unit inaccordance with priority designated by the selfspecifying information,and the part specifying unit specifies a part on the basis of the selfspecifying information.

As described above, by providing each part with the self-specifyinginformation, at the time of specifying the part, the part informationfor comparison including characteristic information can be generated.Further, by designating the kind of part information for comparison orthe generating order, the efficiency of the retrieving process can beimproved. As a result, the efficiency of the specifying process can beimproved, and the accuracy of specification can be improved.

The processing method and condition in the part specifying unit may beset in the part specifying unit or registered as self-specifyinginformation in a database.

For example, a plurality of selectable processing methods are preset inthe part specifying unit and, at a stage that a part is selected, theoptimum processing method may be selected from the processing methods inaccordance with the designation of the self-specifying information inthe attribute data of the part.

Further, for example, not only the selection of the processing method,but also a program of the processing method in the part specifying unitis set as self-specifying information. By obtaining the program, thepart specifying unit may perform the retrieving process, recognizingprocess, and specifying process in accordance with the self-specifyinginformation.

(Set of Parts)

According to the invention of the eighth aspect, as attribute data of aset of parts, identification codes of a plurality of parts constructingthe part set and a combination condition are registered in the database,and when specified parts satisfy the combination condition, thespecifying unit further specifies a part set obtained by combiningspecified parts.

Consequently, even in the case of an object of which whole image is notuniform, a portion can be specified as a set of parts corresponding tothe portion in the object.

An example of the combination condition of parts is a placement relationof parts. Also the part set is suitable for use in recognition of anobject which is constructed by a plurality of blocks and whose wholeshape changes.

(Four-Dimensional Part)

According to the invention of the ninth aspect, the database has, asattribute data of a four-dimensional part modeled on a series ofoperations of an object, a set of three-dimensional shape data in atime-series order of the object.

Thus, the operation itself of an object can be also recognized.

(General Part)

According to the invention of the tenth aspect, the database has, asattribute data of general parts modeled commonly on an object group,attribute data common to parts modeled on the objects of the objectgroup.

With the configuration, part information for comparison of a widepermissible range can be easily generated. For example, in the case ofrecognizing objects whose shapes are different from each other like farmproducts different from industrial products whose shapes arestandardized, the invention is suitable to be used as means forrepresenting the general shape of the objects.

(Narrowing of Parts)

According to the invention of the eleventh aspect, the general parts andparts commonly having the attribute data of the general parts areassociated with each other in the database, the comparison partgenerating unit generates part information for comparison with respectto the general parts, and when the general part is specified by thespecifying unit, the comparison part generating unit generates partinformation with respect to a part associated with the general part.

With the configuration, an object can be efficiently specified.

A processing method of specifying an object in two stages may bedesignated by the self-specifying function in the sixth aspect.

(Capture of Data)

According to the invention of the twelfth aspect, the database capturesdata obtained from a recognized object image as attribute data of aspecified part or replaces the data obtained from a recognized objectimage with a part of attribute data.

As described above, by capturing or replacing attribute data from theobject image, more accurate attribute data can be derived. For example,in the case where a part is specified, it is desirable to capture orreplace attribute data of a portion which does not coincide with therecognized object image.

The attribute data may be captured on the unit basis of, for example,the basic element or composite element in claim 3.

(Narrowing by Grouping Parts)

According to the invention of the thirteenth aspect, a plurality ofparts are grouped for each set situation in the database, and when theinput image corresponds to any of set situations, the comparison partgenerating unit generates the part information for comparison for a partin the group of the corresponding set situation.

By specifying a part in the group of set situation, the parts using forthe retrieving process can be limited. As a result, the efficiency ofthe retrieving process can be increased.

(Narrowing of Coordinate)

According to the invention of the fourteenth aspect, the retrieving unitlimits a retrieval range in the image information for comparison inaccordance with a scene of an input image.

By limiting the retrieval range, the efficiency of the retrievingprocess can be increased. For example, when an object image of a part ofthe image information for comparison is already recognized, theretrieving process can be limited from the relation between the alreadyrecognized object and a part to be retrieved. For instance, in the caseof searching the image information for comparison for a part of a glass,when an image of a table has already been recognized, the retrievalrange may be limited to the area on the table image.

(Specification by Multi-Viewpoint Coincidence)

According to the invention of the fifteenth aspect, a plurality of theinput units obtain input images of the same object from known directionswhich are different from each other, the comparison image generatingunit generates image information for comparison includingtwo-dimensional shape data from each of the input images obtained by theinput units, the comparison part generating unit generates partinformation for comparison having two-dimensional shape data obtained byprojecting three-dimensional shape data of a part into the knowndirections, and the part specifying unit specifies a part in each imageinformation for comparison and confirms that the same part is specifiedin each of the image information for comparison.

In the case when a part can be specified, the same part is specifiedwith respect to the input images in the different directions. Thus, theaccuracy of specification of parts can be largely improved. As a result,the reliability of recognition of an object can be improved.

For example, when the correspondence to the part information in a singlepiece of image information for comparison is not satisfied, as a generalrule, the correspondence to the part information for comparison is notalso satisfied in the image information for comparison in the otherdirections, so that a part is not specified. In contrast, when thecorrespondence to the part information for comparison is satisfied inone piece of image information for comparison, as a general rule, thecorrespondence to the part information for comparison is also satisfiedin the image information for comparison in the other directions, and apart is specified.

At the time of generating two-dimensional comparison image information,for example, data obtained by performing two-dimensional integraltransform on input images of multiple viewpoints by a method such asFourier transform may be used. Also at the time of generatingtwo-dimensional part information for comparison, for example, when it isassumed that a part is placed in the position of the object,two-dimensional integral transform data of the object image of the partobtained by a camera for taking images of the image from a knowndirection may be generated. The part specifying unit obtains thecorresponding relation between two-dimensional integral transform dataand can specify a part when the data coincides with each other.

(Specification by 2D—2D)

According to the invention of the sixteenth aspect, the input unitobtains an input image including an object image photographed from asingle direction, the comparison image generating unit generates imageinformation for comparison including two-dimensional shape data from theinput image, and the comparison part generating unit generates partinformation for comparison having two-dimensional shape data obtained byprojecting the three-dimensional shape data of the part into anarbitrary direction.

The attribute data of a part includes three-dimensional shape data.Consequently, attribute data such as three-dimensional shape data can begiven to part information for comparison projected in a known direction.As a result, at the time of retrieving the image information forcomparison, by the attribute data of the part information forcomparison, a three-dimensional shape, coordinates, and the like of theobject in the corresponding portion can be predicted.

For example, when a part candidate is a desk, a situation in which thedesk is provided upright on a floor can be preferentially retrievedrather than a situation in which the desk floats in the air or asituation in which the desk is inverted. As a result, the retrievalefficiency can be improved.

(Specification by 3D—3D)

According to the invention of the seventeenth aspect, the input unitobtains input images having parallax of the same object from directionswhich are different from each other, the comparison image generatingunit generates image information for comparison includingthree-dimensional shape data from each of the input images, and thecomparison part generating unit generates part information forcomparison having three-dimensional shape data of a part.

Since the three-dimensional shape data of the whole circumference ispreliminarily given to a part, part information for comparison havingthe three-dimensional shape data can be generated. Also from a pluralityof input images, the image information for comparison having thethree-dimensional shape data of a part of the object can be generated asa stereo image by parallax. Consequently, with the three-dimensionalshape of the part information for comparison, the three-dimensionalshape portion in the image information for comparison can be directlyretrieved. As a result, a part can be specified within athree-dimensional coordinate system directly. Thus, a part can bespecified simply with reliability.

At the time of generating three-dimensional image information forcomparison, for example, it may be generated by performingthree-dimensional integral inverse transform on data in a plurality ofdirections obtained by performing two-dimensional integral transform oninput images of an object obtained from a plurality of directions by amethod such as Fourier transform. Also at the time of generatingthree-dimensional part information for comparison, for example, when itis assumed that a part is placed in the position of the object,three-dimensional data of the part to be captured by a camera for takingan image of the part from a known direction may be generated fromattribute data by calculation.

According to the invention of the eighteenth aspect, the part specifyingunit has a settling unit for determining a three-dimensional shape of aspecified part and three-dimensional coordinates indicative of anarrangement relation.

Further, at the time of determining three-dimensional coordinates, dataobtained from an input image, which is not preliminarily included inattribute data of a specified part may be added to the attribute data.

By fixing the parts as described above, not only simple imagerecognition but also more advanced image process and image understandingcan be realized by using the three-dimensional coordinates and attributedata of the specified part. For example, from the three-dimensionalcoordinates, the three-dimensional position relation of partscorresponding to the objects can be derived. Further, for instance, fromthe three-dimensional coordinates and attribute data of each part,information indicative of the relation of parts corresponding to objectsand data necessary for total determination on meaning of the situationshown by the input image is considered to be lead.

(Distinguishing Same Part)

According to the invention of the nineteenth aspect, when the same partis specified with respect to a plurality of different object images bythe part specifying unit, the settling unit adds identifiers which aredifferent from each other to identification codes of the specifiedparts.

As described above, by adding an identifier to an identification code,even in the case where a plurality of objects are of the same kind, theobjects can be recognized individually and distinguished from eachother.

(Trace)

According to the invention of the twentieth aspect, when the input imageis a moving image constructed by a plurality of frames, the partspecifying unit specifies a part with respect to one of the frames andrepeatedly performs only the settling process with respect to the partonce specified on the other frames.

Consequently, once a part is specified, even if the object moves, it isunnecessary to re-perform the recognizing process and specifyingprocess. That is, without changing the identification code of a part,while updating only the position of the object image (for example,coordinate data), the object can be traced. As a result, the partspecification result can be used repeatedly, so that extremely efficienttransfer, recording, and display can be realized.

For example, in input images continuously input such as video images,the position of the same object is displaced continuously. Consequently,in continuous images and the like, the object in a predetermineddeviation range can be sequentially specified as the same part. Thus, itbecomes unnecessary to re-specify the same part each time the inputimage is updated.

(Free Viewpoint)

According to the invention of the twenty-first aspect, the output unitreconstructs a plurality of parts subjected to the settling process inthe part specifying unit and three-dimensional space arrangement of theparts as an image seen from a viewpoint in an arbitrary position anddisplays the result.

Each part has three-dimensional shape data. Consequently, even wheninput images are images obtained only from one direction, with respectto each of parts reconstructed, data of the image seen from an arbitrarydirection can be obtained. As a result, an image showing a state wherethe whole part group is seen from a viewpoint different from that of theinput image can be output.

Consequently, a plurality of objects and their placement relations inthe three-dimensional space can be recreated as a placement relation ofparts modeled on the basis of attribute data of the corresponding parts.

(Camera Calibration and Coupling of Three-dimensional Images)

According to the invention of the twenty-second aspect, the input unitobtains an overlapped portion of three-dimensional spaces in imagecapturing ranges of input images on the basis of an object image in eachof input images of an object whose three-dimensional shape and positionare known, obtained from directions which are different from each other,aligns the overlapped portions so as to coincide with each other on athree-dimensional coordinate system, thereby coupling the images, andobtains a viewpoint position and an angle of view of each of the inputunits.

The specified and settled part has three-dimensional shape data andthree-dimensional coordinate data. Consequently, by coupling theoverlapped portions from object images of a known object, images fromdifferent cameras can be coupled and, simultaneously, the direction ofthe viewpoint to the object can be obtained. Further, the direction ofthe viewpoint in the case of obtaining images of the object from anotherdirection can be also derived. Therefore, the viewpoint direction,viewpoint: position, and angle of view of each of the input units whichobtain images of the same object from multiple directions can beobtained from parts specified and settled with respect to the object.

A plurality of input images are not limited to images obtained at thesame time. For example, also in the case where the input unit movesrelative to the object, if the object does not move relative to thecoordinate system as a reference, images can be similarly coupled andthe viewpoint position and the angle of view can be obtained.

(Communication)

In order to achieve the second object in addition to the first object,according to the invention of the twenty-third aspect, the informationconverting system further includes: a transmitting unit for transmittingan identification code output from the output unit to a communicationline; a receiving unit for receiving the identification code; areception-side database in which the identification code and attributedata are associated with each other and registered; and a reconstructingunit for searching the reception-side database for attribute data of apart corresponding to the identification code and outputting theattribute data.

With the configuration, by transferring the identification code andposition information of a part specified on the transmission side, animage obtained by converting the object image in the input image to apart can be reconstructed on the reception side. As a result, it becomesunnecessary to transfer image data of the object. Thus, the transmissionamount is largely reduced, so that high-speed transmission can berealized, and a load on the line can be lessened.

(Different Part Storage)

According to the invention of the twenty-fourth aspect,three-dimensional shape data of parts of the same identification code inthe database on a transmission side and the reception-side database aredifferent from each other.

A part registered in the database on the reception side may or may notcoincide with a part registered in the database on the transmissionside.

For example, in the case of transmitting only information of theplacement state of objects or the like quickly, the data in thedatabases do not always have to be the same. Also for example, for easyexplanation, although it is different from an object, the object may bereproduced as a part symbolically representing the object by animationor illustration.

(Analysis Information)

According to the invention of the twentyfifth aspect, the informationconverting system further includes an analysis information generatingunit for combining attribute data of a plurality of parts specified bythe part specifying unit to thereby generate analysis informationregarding a group of the parts.

With the configuration, not only each object but also the state of thewhole object group can be recognized. For example, by combining weightdata in attribute data of objects, the total weight of all the objectscan be generated as analysis attribute information.

Attribute data to be combined is not limited to those of the same kind.Attribute data of different kinds of different parts may be alsocombined with each other.

Further, when analysis attribute information is generated by usingattribute data which is not included in an input image, information(such as the date of manufacture) which is difficult to be recognized ordetermined by a human being only from input images can be alsoautomatically recognized and determined.

It is desirable to provide an item selecting unit for selecting a way ofcombining parts and attribute data used for generating analysisinformation in accordance with analysis information to be generated. Byproviding the item selecting unit, the combining way adapted to apurpose can be selected and used. As a result, the informationconverting system can be used as a general system which is not limitedto a specific use.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram for explaining the configuration ofan information converting system of a first embodiment.

FIG. 2 is a table for explaining an example of an identification codeand attribute data in a database.

FIG. 3 is a flowchart for explaining the operation of the informationconverting system of the first embodiment.

FIG. 4 is a schematic diagram showing a state in which a plurality ofparts are grouped and registered in a database.

FIG. 5A is a perspective view showing objects of information conversionin the first embodiment and FIG. 5B is a schematic diagram of an inputimage.

FIG. 6 is a block diagram for explaining specification of a part byusing two-dimensional image information for comparison andtwo-dimensional part information for comparison.

FIG. 7A is a schematic diagram of parts corresponding to objects shownin FIG. 5B and FIG. 7B is a display screen showing a group of recognizedparts.

FIG. 8 is a flowchart for explaining processes of a part specifyingunit.

FIG. 9 is a table for explaining an example of coordinate codes of thefirst embodiment.

FIG. 10A is a display screen showing a state that a reconstructed partgroup is seen from a side viewpoint and FIG. 10B is a display screenshowing a state that the reconstructed part group is seen from an upperviewpoint.

FIGS. 11A and 11B are schematic diagrams of part information forcomparison of basic elements.

FIGS. 12A and 12B are explanatory diagrams of an element extractingfilter.

FIGS. 13A and 13B are schematic diagrams for explaining deformation of abasic element.

FIGS. 14A and 14B show examples of the part information for comparison.

FIG. 15 is a schematic diagram of image information for comparisondecomposed to basic elements.

FIG. 16A is a schematic diagram of part information for comparison ofcomposite elements of a characteristic portion, and FIG. 16B is aschematic diagram of a portion of pixels for comparison decomposed tobasic elements.

FIG. 17A is a list of vector display of corners and lines on the partside and FIG. 17B is a list showing the corresponding relation betweenthe corner and line on the part side.

FIG. 18 is a list of vector display of corners and lines on the inputside.

FIG. 19 is a list showing the corresponding relation between the cornersand lines on the input side.

FIG. 20 is a schematic diagram showing that a desk portion in an inputimage is specified.

FIG. 21 shows input images in a third embodiment.

FIG. 22A shows silhouettes as general parts and FIG. 22B shows anelement extracting filter of a silhouette.

FIGS. 23A to 23C are explanatory diagrams of a process of taking inputinformation into attribute data.

FIG. 24 is a block diagram for explaining a fourth embodiment.

FIG. 25 is an explanatory diagram showing the placement relation of anobject and a camera in the fourth embodiment.

FIGS. 26A to 26C show input images in the fourth embodiment.

FIG. 27 is a block diagram for explaining the configuration of aninformation converting system of a fifth embodiment.

FIG. 28 is a table for explaining an example of identification codes andattribute data registered in a reception-side database.

FIG. 29A is a schematic diagram of modeled parts in the fifthembodiment, and FIG. 29B is a display screen showing a reconstructedpart group.

FIG. 30 is a detailed functional block diagram of an informationconverting system of the fifth embodiment.

FIG. 31 is a functional block diagram continued from the functionalblock diagram of FIG. 30.

FIG. 32 is an explanatory diagram showing, as a model, an algorithm forperforming comparison and recognition with a database regarding an inputimage and parts in a correlation function computing means for partretrieval shown in FIG. 30.

FIG. 33 is an explanatory diagram showing, as a model, an informationprocess of a system configuration in an information converting systemshown in FIGS. 30 and 31.

FIG. 34 is a block diagram for explaining the configuration of aninformation converting system of a sixth embodiment.

FIG. 35 is a flowchart for explaining the operation of the informationconverting system of the sixth embodiment.

FIG. 36A is a schematic diagram of an input image in the sixthembodiment and FIG. 36B is a schematic diagram of a registered vehicle.

FIG. 37 is a block diagram for explaining the configuration of aninformation converting system of a seventh embodiment.

FIG. 38 is a flowchart for explaining the operation of the informationconverting system of the seventh embodiment.

FIG. 39A is a schematic diagram of an object in the seventh embodimentand FIG. 39B is a schematic diagram showing modeled parts.

FIG. 40 is a block diagram for explaining the configuration of aninformation converting system of eighth and ninth embodiments.

FIG. 41 is a flowchart for explaining the operation of the informationconverting system in the eighth embodiment.

FIG. 42 is a schematic diagram of objects in the eighth and ninthembodiments.

FIG. 43 is a flowchart for explaining the operation of the informationconverting system in the ninth embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the invention will be described hereinbelowwith reference to the drawings. However, the invention is not limited tothe embodiments.

First Embodiment

A first embodiment will be described with reference to FIGS. 1 to 10.

1. Configuration of Information Converting System

FIG. 1 is a functional block diagram of an information converting systemof the first embodiment. As shown in FIG. 1, the information convertingsystem of the first embodiment has a database (DB) 300, a comparisonpart generating unit 400, an input unit 100, a comparison imagegenerating unit 200, a part specifying unit 500, and an output unit 600.

(1) Database

First, the database 300 will be described.

In the database 300, parts as models of objects are registered. To eachpart, various attribute data such as data indicative of athree-dimensional shape of an object, characteristics, properties, andcircumstances is given. The attribute data of each part is associatedwith an identification code of the part and registered in the database300.

It is generally desirable that the number of parts to be registered inthe database 300 is as large as possible. However, in the case whereobjects to be recognized are limited, it is sufficient to register onlyparts corresponding to the limited objects.

Generally, it is desirable that the kinds of attribute data are variousas much as possible. However, depending on the purpose of recognizing anobject or the kind of an object, the kinds of attribute data may belimited.

FIG. 2 shows an example of identification codes and attribute dataregistered in the database 300. As shown in FIG. 2, in the database 300,attribute data such as the name, three-dimensional shape, color, and useof a part is stored in association with an identification code (ID) ofthe part. The attribute data of three-dimensional shape and color isstored as numerical value data.

In the database 300, as attribute data of a part set M10, the ID codesof a plurality of parts M1 constructing the part set M10 and acombination condition may be registered. FIG. 3A schematically shows theconcept of the part set M10.

An example of the part set is a human's face. In this case, parts of aface may be registered as a part set of combined parts such as eyes,mouth, and nose. The positional relations of the parts of eyes, mouth,and the like may be registered as a combination condition.

Another example of the part set is a car. In this case, as a part set ofcombined parts of tires, doors, and the like, parts modeled on the carmay be registered. The positional relations of the parts of tires,doors, and the like may be used as a combination condition.

Further, in the database 300, as attribute data of a four-dimensionalpart modeled on a series of operations of an object, a set ofthree-dimensional shape data of the object in a time-series order may beregistered. FIG. 3B schematically shows the concept of afour-dimensional part M20. By using the four-dimensional part M20, theoperation itself of an object, for example, an operation of running of ahuman being or a gesture can be also recognized.

In the database 300, as attribute data of general parts commonly modeledon a group of objects, attribute data common to parts modeled on theobjects of the group may be registered.

It is suitable to use general parts on an occasion of recognizingobjects having different shapes such as farm products.

An object may be specified once by using the general parts and furtherspecified by each part.

In the database (part storage) 300, a plurality of parts may be groupedand registered for each set circumstance.

FIG. 4 schematically shows an example of a plurality of parts Ml groupedin a related commodity shelf 301. In the related commodity shelf 301,for example, when a set circumstance is “indoor,” parts of furnituresuch as “desk” and “chair” may be grouped and registered.

By grouping parts in such a manner, parts used for a retrieving processcan be limited to parts in a group corresponding to the setcircumstance. As a result, the retrieving process can be performed moreefficiently.

In the embodiment, attribute data of each of parts registered in adatabase includes self-specifying information which instructs aprocessing method of specifying the part itself.

By the self-specifying information, the kind of data used for partinformation for comparison in the attribute data of the part isdesignated and the order of the part information for comparison used forretrieval is also designated.

The processing method in the part specifying unit may be preset in thepart specifying unit 500 or set as self-specifying information. In thecase of setting the processing method as self-specifying information,the following two examples can be considered.

As a first example, a plurality of selectable processing methods arepreset in the part specifying unit 500, and the part specifying unit 500is allowed to select the optimum processing method from the processingmethods.

As a second example, a program of the processing method in the partspecifying unit 500 is set as self-specifying information, and theprogram is sent to the part specifying unit 500 and executed there.

Since three-dimensional shape data of the whole circumference ispreliminarily given to a part, the comparison part generating unit 400may generate part information for comparison including thethree-dimensional shape data of the part. In this case, for example,when it is assumed that a part is placed in the position of the object,three-dimensional data of parts to be taken by a camera for acquiringimages of the part from a known direction may be generated bycalculation from the attribute data.

(2) Comparison Part Generating Unit

The comparison part generating unit 400 will now be described.

The comparison part generating unit 400 generates one or plural partinformation for comparison from attribute data on a part-by-part basis.For example, as part information for comparison, two-dimensional imagesobtained by projecting a part having three-dimensional shape data tovarious directions may be generated. Alternately, for example, as thepart information for comparison, integral transform data may begenerated by a method such as Fourier transform from the attribute dataof a part.

The range and order of candidate parts for generating the partinformation for comparison may be preset or instructed by the operator.

When a part has self-specifying information, the comparison partgenerating unit 400 generates the part information for comparisondesignated by the self-specifying information. The comparison partgenerating unit 400 outputs the part information for comparison to thepart specifying unit in a priority order designated by theself-specifying information.

(3) Input Unit

The input unit 100 will now be described.

The input unit 100 acquires an input image including an image of anobject. In the first embodiment, the input unit 100 takes the form of astereo camera. By the stereo camera, as input images, pseudothree-dimensional images having parallax are obtained by taking imagesof an object from different directions.

(4) Comparison Image Generating Unit

The comparison image generating unit 200 will now be described.

The comparison image generating unit 200 performs an imaging process onan input image to thereby generate image information for comparisonwhich has not been subjected to an advanced imaging process of, forexample, clipping an image of an object.

In the embodiment, the comparison image generating unit 200 generatesimage information for comparison having data of the same kind as that ofdata of the part information for comparison. For example, when the partinformation for comparison is a two-dimensional image, image informationfor comparison of a two-dimensional image is generated. For example,when only an outline of a two-dimensional image is extracted as partinformation for comparison, image information for comparison obtained byextracting only an outline from an input image is generated. Forinstance, as the image information for comparison, integral transformdata may be generated from an input image by a method such as Fouriertransform.

The comparison image generating unit 200 may generate image informationfor comparison including three-dimensional shape data from a stereoimage obtained by the input unit 100. In this case, for example, theimage information for comparison may be generated by three-dimensionalintegrate inverse transform from data obtained from various directionsby performing two-dimensional integral transformation on input images ofan object captured from a plurality of directions by a method such asFourier transform.

Although the image information for comparison is generated in accordancewith the kind of data of the part information for comparison in theembodiment, in the invention, the comparison part generating unit 400may generate part information for comparison in accordance with the kindof data of the image information for comparison.

(5) Part Specifying Unit

The part specifying unit 500 will now be described.

The part specifying unit 500 specifies parts corresponding to an imageof an object by using the part information for comparison and imageinformation for comparison having data of the same kind.

For this purpose, the part specifying unit 500 has a retrieving unit510, a recognizing unit 520, a specifying unit 530 and a settling unit540.

In the embodiment, each object image is not clipped from an input imageand is not collated with a part.

(5-1) Retrieving Unit

The retrieving unit 510 retrieves a corresponding portion whichcorresponds to the part information for comparison from the imageinformation for comparison sequentially with respect to one or pluralpart information for comparison of one or plurality parts. Theretrieving unit 510 determines whether a portion corresponding to atleast a part of the part information for comparison exists or not in theimage information for comparison.

At the time of retrieval of the corresponding portion, a portion in anyof the image information for comparison and part information forcomparison do not have to be completely matched. For example, it issufficient to determine a corresponding portion when the portioncoincides with a portion of a predetermined ratio or larger out ofelements of the part information for comparison.

(5-2) Recognizing Unit

The recognizing unit 520 recognizes a corresponding part in the imageinformation for comparison as an object image. In such a manner, withoutclipping an image of the object from an input image, the area of theobject image in the input image can be defined.

In the case of performing a process of retrieving another part after anypart in the image information for comparison is recognized as an imageof the object, it is sufficient to search the remaining portion of thecomparison image.

(5-3) Specifying Unit

The specifying unit 530 specifies a part having the part information forcomparison as a part corresponding to the object's image, and obtainsthe identification code (ID) and attribute data of the part from thedatabase 300.

By the operation, the image information of the object is converted todata of the part. Therefore, the object (“what is the object”) is notrecognized by a human being but can be automatically recognized by thecomputer.

(5-4) Settling Unit

The settling unit 540 determines the position of the specified part bythe position of the recognized object's image. Further, the settlingunit 540 determines the placement direction of the part in thedetermined position on the basis of the data of the comparison partinformation corresponding to the object image.

Depth information in the position information of the object is obtainedas, for example, a parallax amount of a stereo image. Also in the casethat an image is not a stereo image, by specifying parts, the relativepositional relation of objects may be obtained as a result.

For example, in the case where a table is placed on a horizontal floorand the floor, the table, and a glass placed on the table arerecognized, each of a floor as a part, a table as a part, and a glass asa part has three-dimensional shape data. Consequently, even in the casewhere there is no parallax, the positional relations among the floor asa part, the table as a part, and the glass as a part can be narroweddown. For example, from the existence of gravity and rational assumptionthat the table and glass do not float in the air, the positionalrelations of the parts can be narrowed down to their horizontal plane.Within the range of the horizontal plane, a portion in an image matchingany of the parts can be retrieved. When there is a match, the part isspecified. Further, by executing backward calculation from the size anddirection of the three-dimensional shape of the specified part,three-dimensional positions of the table and glass can be obtained.

The information of the placement direction of a part is usually includedin data of the part information for comparison. For example, in the caseof generating the part information for comparison having two-dimensionalshape data obtained by two-dimensionally projecting three-dimensionalshape data of a part, the part information for comparison includesinformation of the projection direction. Consequently, on the basis ofthe information of the projection direction of the part information forcomparison having the corresponding part which is found, the placementdirection of the part can be determined.

The settling unit makes final determination on not only the positionaldirection obtained from the object but also, as necessary, informationregarding the five senses such as the shape, color, sound, smell, andhardness preliminarily given as attribute data to a part and, asfurther, information created by a human being such as the date ofmanufacture.

There is a case such that the same part is specified with respect to aplurality of different object images. In this case, since the objectshave different position coordinates, they can be separated from eachother. The settling unit 540 adds identifiers which are different fromeach other as the identification codes of specified parts. Thus, theobjects for which the same part is specified can be individuallyrecognized.

As a method of describing the identifier, an arbitrary suitable methodcan be used. For example, an alphabetical character or a number may beadded to the end of an identification code.

(5-4-1) Trace

When input images are continuously input as video images, the positionof the same object is continuously displaced. Because of this, in framesof the continuous input images, the object within a predetermineddeviation range can be sequentially specified as the same part. In thiscase, once the part is specified in one frame, it is sufficient toperform only a settling process on the other frames. That is, withoutchanging the identification code of a part, while repeatedly updatingthe position of the part by the settling unit 540, the object can betraced.

(6) Output Unit

The output unit 600 outputs the identification code and at least a partof the attribute data of the specified part as a result of therecognizing process, specifying process, and settling process on theobject's image.

The output unit 600 may reconstruct a plurality of parts and spatialplacement of the parts as an image seen from a viewpoint in an arbitraryposition and display the resultant. In this case, a plurality of objectsand the placement relation of them can be reproduced as a placementrelation of modeled parts on the basis of the attribute data ofcorresponding parts. By the above, computer graphics (CG) in which thestate of real space is reflected can be easily generated.

The output unit 600 can output not only CG but also information such assound and the data of manufacture included in the attribute data of apart as necessary.

Parts and their placement relation are determined by the settling uniton the basis of the identification code of the specified part andposition data added to the attribute data and the resultant is output.

As described above, according to the embodiment, the object can beconverted to the part to which the attribute data is given and theresultant can be output. As a result, when data is included in attributedata preliminarily given to the specified part, even the data is notincluded in an input image, the data can be output. For example, theshape of a back side portion of a desk and weight information of thedesk and a chair, which does not appear in the input image can be alsooutput as attribute data.

Further, by specifying parts, not only the name of the object is simplyrecognized but also a process after the parts are specified can beperformed by replacing parts with the parts whose attribute data isregistered. Thus, more advanced image recognition and imageunderstanding can be realized. As a result, the possibility of applyingthe invention as a basic technique of the information processingtechnique such as artificial intelligence to various aspects isexpected.

2. Example of Process

An example of recognizing furniture or the like in a room shown in FIG.5A will now be described.

FIG. 5A shows a state where a desk 31, a chair 32, a fire extinguisher33, and a dust box 34 as objects are placed in a room.

(1) Acquisition of Input Image

First, the furniture and so on shown in FIG. 5A is photographed by thestereo camera as the input unit 100 to obtain an input image Im2 (FIG.6).

Stereo images 40 a and 40 b obtained are shown in FIG. 5B. In each ofthe stereo images 40 a and 40 b, an image 41 of the desk, an image 42 ofthe chair, an image 43 of the fire extinguisher, and an image 44 of thedust box are included with parallax.

In FIG. 5B, in order to emphasize that the input image is image data,the image 41 of the desk and the like have hatching for convenience.

(2) Generation of Image Information for Comparison

Next, the stereo image 40 a or 40 b is subjected to an imaging processby the comparison image generating unit 200 to thereby generate imageinformation Im2 for comparison from which the object's image is notclipped (FIG. 6).

In the embodiment, by a conventional known arbitrary suitable method,outlines in an input image are extracted to generate the imageinformation for comparison. In the image information for comparison, theoutlines of images of neighboring objects are continuous and the imageof each object is not clipped. For example, in the example shown in FIG.5B, the outline of the desk 41 and that of the chair 42 are continuous.The outline of the fire extinguisher 43 and that of the dust box 44 arealso continuous.

(3) Generation of Part Information for Comparison

In the comparison part generating unit 400, as shown in FIG. 6,two-dimensional images obtained by projecting a part Ml havingthree-dimensional shape data to multiple directions are generated aspart information M2 for comparison.

FIG. 7A shows an example of the part information M2 for comparison of adesk 51 as a part, a chair 52 as a part, a fire extinguisher 53 as apart, and a dust box 54 as a part. Although FIG. 7A shows one piece ofpart information for comparison for each part, in reality, a pluralityof pieces of part information for comparison are generated per part. Forexample, as part information for comparison of the desk 51 as a part,plural pieces of part information for comparison obtained by seeing thedesk from various directions are generated. Also for each of parts otherthan the furniture and the like shown in FIG. 7A, part information forcomparison is generated.

Either the image information for comparison or the part information forcomparison may be generated first. The part information for comparisonmay be either generated in advance for each of parts or generated eachtime in accordance with necessity.

(4) Specification of Part

By the part specifying unit 500, parts corresponding to the objectimages 41 to 44 are specified by using the part information M2 forcomparison and the image information Im2 for comparison having the samekind of data.

(4-1) Retrieving Process

With reference to the flowchart of FIG. 3, the processes performed bythe part specifying unit 500 will be described.

First, the retrieving unit 510 obtains a comparison part from thecomparison part generating unit 400 (step S1 in FIG. 8).

Subsequently, the retrieving unit 510 retrieves the corresponding partwhich corresponds to the part information M2 for comparison from theimage information Im2 for comparison (step S2 in FIG. 8).

Further, the retrieving unit 510 determines whether a portioncorresponding to the part information M2 for comparison exists in theimage information Im2 for comparison or not (step S3 in FIG. 8).

The retrieving unit 510 repeats the processes of steps S1 to S3 withrespect to sequentially one or plural part information for comparison ofone or plural parts until a corresponding portion is found (steps S8 toS10 in FIG. 8). When a part corresponds to the information, as shown inFIG. 5, among the plural pieces of part information M2 for comparisongenerated from the part, part information M2 for comparison astwo-dimensional shape data seen from any of the directions correspondsto the portion of the object's image in the image information forcomparison.

When a corresponding portion is not found after all of the predeterminedpieces of the part information for comparison of a predetermined partare searched, it is determined that there is no corresponding part (stepS7 in FIG. 8), and the process is finished.

(4-2) Recognizing Process

When the corresponding portion is found, the recognizing unit 520recognizes the corresponding portion in the image information forcomparison as an object's image (step S4 in FIG. 8). In such a manner,without individually clipping the object's image from an input image,the area of the object's image in the input image can be defined.

(4-3) Specifying Process

Subsequently, the specifying unit 530 specifies a part having the partinformation for comparison as a part corresponding to the object's image(step S5 in FIG. 8). The identification code (ID) and attribute data ofthe part are obtained from the database 300.

As a general rule, the processes of steps S1 to S5 are repeated untilall the object's images of the predetermined portion in the imageinformation for comparison are specified (step S6 in FIG. 8).

(4-4) Settling Process

In the embodiment, simultaneously with the specifying process (step S5),by the settling unit 540, the position of the specified part isdetermined by the position of the recognized object's image. Further,the settling unit 540 determines the direction of placement of the partin the determined position on the basis of the data of the partinformation for comparison corresponding to the object's image.

FIG. 9 shows an example of position coordinate data. In the exampleshown in FIG. 9, as position coordinate data of each part, XYZ-axescoordinates of the position of each part and XYZ-axes coordinates of thedirection of each part are generated.

At the time of specifying a part, image information such as hue andshading of an object's image may be taken into attribute data of a part.Data of a part of input information and attribute data peculiar to thepart may be combined and output. For example, sound information orinformation of brightness in input information may be reproduced as itis and output.

Data of the shapes and the like of naturally created matters, which aredifferent from each other, may be added to the attribute data of parts.With respect to a matter whose shape is deformed such as a crashedguardrail, the attribute data may be replaced with data of the deformedshape.

Referring to FIG. 23, an example of replacing data in input image withattribute data of a part will be described.

FIG. 23A shows an input image of a bonsai, a miniature potted tree. FIG.23B shows a bonsai as a specified part. With respect to the bonsai inthe input image and the bonsai as a part, the shape of an image 46 of apot and that of a pot 56 of the part almost coincide with each other.However, the shape of an image 47 of the tree portion and that of a treeportion 57 of the part are different from each other. Consequently, atree portion 57 a as a part is generated from the input image, therebygenerating a bonsai as a new part. FIG. 23C shows the bonsai as anupdated part.

(6) Output

FIG. 7B shows the parts specified in such a manner.

As shown in FIG. 7B, as a part corresponding to the portion of the image42 of the chair, the chair 52 as a part is specified. As a partcorresponding to the portion of the image 43 of the fire extinguisher,the fire extinguisher 53 as a part is specified. As a part correspondingto the portion of the image 41 of the desk, the desk 51 as a part isspecified. As a part corresponding to the portion of the image 44 of thedust box, the dust box 54 as a part is specified.

When the identification codes and positional information of the parts 51to 54 are stored, by using the attribute data stored in the database300, a display image 50 shown in FIG. 7B can be easily constructed. Itis therefore unnecessary to store the stereo images 40 a and 40 b shownin FIG. 5B. As a result, the storage amount of storage informationregarding an object can be largely reduced.

(6-1) Free-Viewpoint Display

Since each part has tree-dimensional shape data, even an input image isan image obtained only from one direction, data of the wholecircumference can be obtained with respect to each of the parts 51 to54. As a result, an image showing a state where the whole group of partsis seen from a viewpoint different from that of the input image can beoutput. The placement relation in three-dimensional space of the objectscan be reconstructed as the placement relation of the parts.

For example, as shown in a reconstructed image 50 a of FIG. 10A, theplacement relation of the group of the parts 51 to 54 seen from a sidedirection of the desk 51 as a part can be presented. As shown in areconstructed image 50 b of FIG. 10B, the placement relation of thegroup of the parts 51 to 54 seen from the above can be also presented.

Second Embodiment

Referring to FIGS. 11 to 20, a second embodiment will be described.

The configuration of an information converting system in the secondembodiment is basically the same as that in the first embodiment shownin FIG. 1.

(1) Part Information for Comparison

However, in the second embodiment, different from the first embodiment,as part information for comparison, the comparison part generating unit400 decomposes the attribute data of a part into basic elements such asoutlines to thereby generate the individual basic elements or acomposite element obtained by combining a plurality of basic elements.

The basic elements include all the elements which can be extracted froman input image as elements constructing the input image. Examples of thebasic elements are a line segment of a straight line portion, a curveportion, and a corner portion of an outline. The corner portion as abasic element includes a right-angle portion and a T-shaped portion ofan outline. The basic elements may be drawn by, for example, vectors.

Examples of the composite element are a plane specified by a pluralityof straight line portions and corner portions, a curved surface, asurface of the same texture, a surface of a continuous hue, and a groupof lines at infinity as a set of line segments which are converged tothe same point.

It is desirable to give an element recognition code to each of the basicelement and composite element. As a result, the input image is describedby the element recognition codes.

FIG. 11 shows an example of part information for comparison decomposedto basic elements. FIG. 11A schematically shows a state where theoutline of the desk 51 as a part is decomposed into line segments ofstraight line portions and corner portions. FIG. 11B shows partinformation for comparison obtained by extracting only the main basicelements.

(1-1) Element Extracting Filter

Further, each basic element is described by an element extractingfilter. The element extracting filter takes the form of atwo-dimensional matrix or three-dimensional matrix in which a high pointis given to a pixel which coincides with the basic element or compositeelement and a low point is given to a pixel apart from the shape of theelement.

FIG. 12A shows an example of the two-dimensional element matrix. Theelement matrix corresponds to the basic element of the corner portion ofan L-letter shape. “5” is given to a portion which coincides with theshape of the basic element and the points decrease step by step like“3,” “1,” “1,” and “−3” as the distance from the L-letter shapeincreases.

The values and distribution of the points can be arbitrarily set.

In the case of using the element extracting filter, the retrieving unitretrieves, as a corresponding portion, a portion in which the totalpoint of the pixels that coincide with the basic element or compositeelement of the image information for comparison is the highest.

For example, FIG. 12B shows a state where an L-letter portion C in theoutline in the image information for comparison overlaps with theelement extracting filter. When the L-letter portion accuratelycoincides with the basic element of the element extracting filter, thetotal point becomes “5×15=275.” In contrast, the total point of the caseshown in FIG. 12B is “1×3+3×3+5×5+3×4=49.” By turning or moving theelement extracting filter on the image information for comparison, theportion in which the total point is the highest may be retrieved.

Further, by using the element extracting filter, the permissible rangeat the time of the retrieving process can be widened. FIG. 13A shows thepart information for comparison obtained by decomposing the outline of acar into basic elements. In FIG. 13A, the basic element of each of astraight line portion and a curve portion in the outline is indicated bya double-headed arrow.

FIG. 13B shows a state where a car of a similar shape can be alsoretrieved by giving a permissible range to the length of each basicelement.

As the part information for comparison of the basic element or compositeelement, not only each of the above-described line segments of theoutline but also an outer outline signal shown in FIG. 14A or asilhouette signal shown in FIG. 14B can be also used.

In the second embodiment, the comparison image generating unit 200extracts basic elements of the outline or the like as image informationfor comparison and generates a set of the basic elements or compositeelements, and the retrieving unit retrieves a portion corresponding tothe basic element or composite element of a part from the imageinformation for comparison.

(2) Image Information for Comparison

FIG. 15 shows an example of the part information for comparisonrepresented by a set of basic elements of an outline or the like. FIG.15 schematically shows a state where the outline of each of the desk 41as a part, chair 42 as a part, fire extinguisher 43 as a part, and dustbox 44 as a part is decomposed to line segments of straight lineportions and corner portions.

At the time of retrieval, in the second embodiment, the comparison partgenerating unit 400 further generates, as part information forcomparison, a composite element of only a characteristic portion of theattribute data of a part. For example, in the case where the desk 51 asa part is decomposed to basic elements, as shown in FIG. 16A, acomposite element is generated only by basic elements defining a topboard. The composite element of the top board is defined by four cornerportions and four straight line portions sharing the visual point withthe corner portions. Element identification codes (c-1 to c-4) are givento the corner portions and element identification codes (l-1 to l-4) aregiven to the straight line portions.

The composite element of the top board is specified only by the couplingrelation of the basic elements. Specifically, information of thedirection, distance, and shape of each basic element is erased and onlythe order of coupling the basic elements has meaning.

In the second embodiment, the retrieving unit 510 searches thecomparison image shown in FIG. 15 for a portion corresponding to thecomposite element of the top board on the unit basis of the basicelement or composite element.

An example of retrieving the basic elements corresponding to thecomposite element of the top board (corners c-1 to c-4 and lines l-1 tol-4) from basic elements (corners c-01 to c-11 and lines 1-01 to 1-07)of the outline of the image 41 of the desk among the basic elements ofthe outline shown in FIG. 15 will be described.

FIG. 17A shows a list of vector display of the corners and lines on thepart side. FIG. 17B shows the corresponding relation between the cornersand lines on the part side. FIG. 17B shows that the corners and linesshare vectors and the coupling order forms a loop.

FIG. 18 shows a list of vector display of the corners and lines on thepart side. FIG. 19 shows corners and lines having the same couplingrelation as that of the composite element of a transition differenceamong the corners and lines on the part side illustrated in FIG. 18.Among the basic elements shown in FIG. 16B, a portion defined by fourcorners (c-03 to c-06) and four lines (1-04 to 1-07) is determined to bea corresponding portion.

The correspondence relation of the corners and lines may not beperfectly coincided. For example, when corners and lines of apredetermined ratio or higher correspond, a portion defined by them maybe determined as a corresponding portion.

Subsequently, after the corresponding portion to the basic element orcomposite element of the characteristic portion is retrieved, therecognizing unit 520 detects the correspondence between thecorresponding portion and the basic element or composite element out ofthe characteristic portion of the same part, and recognizes thecorresponding portion as an object image. Concretely, when a portioncorresponding to the composite element of the top board is found, therecognizing unit 520 further detects that the basic element or compositeelement out of the top board portion of the same part shown in FIG. 16Balso corresponds, and recognizes the corresponding portion as the objectimage of the desk.

(3) Specifying Process

Further, the specifying unit 530 obtains the direction of the top boardfrom the shape of the detected top board portion. The specifying unit530 further obtains the direction of the desk, confirms that theoutline, silhouette, and hue of the part correspond to the object image,and specifies the part. FIG. 20 schematically shows a state where onlythe desk portion is specified in the input image.

As described above, by the basic elements or composite element of thecharacteristic portion of the part, the part can be specified byefficiently performing the retrieving process.

It is desirable to register the method of designating and retrievingpart information for comparison in the second embodiment asself-specifying information into the attribute data of a part.

Third Embodiment

Referring to FIGS. 21 and 22, a third embodiment will be described.

The configuration of an information converting system in the thirdembodiment is basically the same as that of the first embodiment shownin FIG. 1.

In the second embodiment, however, different from the first embodiment,general parts of human beings are registered in the database 300. Ageneralized part is obtained by giving common attribute data to partsmodeled on objects in a group as attribute data of a general partcommonly modeled on the object group. In this case, as general parts ofhuman beings, as shown in FIG. 22A, various silhouettes are given asattribute data.

In the third embodiment, as shown in FIG. 21A, an image 45 of a humanbeing is included in the input image. In this case, with respect to thedesk image 41 and the chair image 42, the desk 51 as a part and thechair 52 as a part can be specified in a manner similar to the foregoingfirst or second embodiment. After those parts are specified, asschematically shown in FIG. 21B, only the portion of the human beingimage 45 remains as an unspecified portion.

In the third embodiment, whether the silhouette of the portioncorresponds to any of the silhouettes of the general parts or not isdetermined. For the determination, it is preferable to use a silhouetteelement extracting filter (element operator) as shown in FIG. 22B.

In FIG. 22B, pixels in the element extracting filter are not shown.

In the element extracting filter of FIG. 22B, point “5” is given to apixel in a portion which coincides with the silhouette of the generalpart. Point “3” is given to a pixel near the silhouette. Point “−1” isgiven to a pixel in a portion apart from the silhouette.

The kind and position of the element extracting filter in which thetotal point of the pixels coinciding with the silhouette of theunspecified portion is the highest are obtained, thereby specifying thegeneral part.

When a general part is specified, as necessary, a concrete part relatedto the general part may be specified. By specifying the object at twostages, the object can be efficiently specified.

Fourth Embodiment

Referring to FIGS. 24 to 26, a fourth embodiment will be described.

The configuration of an information converting system in the fourthembodiment is basically the same as that in the first embodiment shownin FIG. 1.

However, in the fourth embodiment, different from the first embodiment,a plurality of input units 100 obtain input images Im1 of the sameobject photographed from known directions which are different from eachother. FIG. 25 shows a state where images of a chair 48 as an object areacquired by cameras 100 a to 100 c in three directions which aredifferent from each other. FIGS. 26A to 26C show input images obtainedby the cameras 100 a to 100 c, respectively.

The comparison image generating unit 200 generates comparison imageinformation Im2 including two-dimensional shape data from the inputimages obtained by the input units 100.

On the other hand, the comparison part generating unit 400 generates thecomparison part information M2 having two-dimensional shape dataobtained by projecting three-dimensional shape data of a part M1 into aplurality of known directions which are different from each other.

The part specifying unit 500 specifies the part M1 for each of the imageinformation Im2 for comparison. In this case, since the same chair imageis included in all of the image information for comparison, the samepart is supposed to be specified. Therefore, the part specifying unit500 confirms that the chair as a part is specified for each of the imageinformation Im2 for comparison.

When the same part can be specified for the chair seen from theplurality of directions as described above, the precision ofspecification of a part can be improved. As a result, the reliability ofrecognition of the object can be improved.

The input units 100 a to 100 c can obtain the overlap ofthree-dimensional spaces in the acquisition range of input images andthe viewpoint positions of the input units on the basis of the objectimages in the input images of the object of which three-dimensionalshape and position are known, obtained from different directions. Thespecified and settled part has three-dimensional shape data andthree-dimensional coordinate data. Consequently, from the object imageobtained by photographing a known object, the direction of the viewpointfor the object can be derived. Further, the direction of the viewpointof the case where the object is photographed from another direction canbe also obtained. Therefore, the viewpoint direction and the viewpointposition of each of the input units for acquiring images of the sameobject from various directions can be obtained by parts specified andsettled with respect to the object. For example, as shown in FIG. 26B,by providing three markers P1 to P3 in known positions, the position ofthe camera 100 b can be obtained according to how the markers are seen.

A plurality of input images are not limited to those acquired at thesame time. For example, also in the case where images of the object arecaptured from directions which are different from each other whilesequentially moving a single input unit, the viewpoint position can beobtained similarly.

Fifth Embodiment

Referring to FIGS. 27 to 29, a fifth embodiment of the invention willnow be described.

First, with reference to the functional block diagram of FIG. 27, theconfiguration of an information converting system of the fifthembodiment will be described. As shown in FIG. 27, the informationconverting system of the fifth embodiment is separated into atransmission side and a reception side.

The information converting system of the fifth embodiment has, inaddition to the configuration of the foregoing first embodiment, atransmitting unit 810, a receiving unit 820, a reception-side database310, and a reconstructing unit 900.

The transmitting unit 810 transmits an identification code output fromthe output unit 600 to a communication line 830. The receiving unit 802receives the identification code. In the reception-side database 310, anidentification code and attribute data are associated with each otherand registered. The reconstructing unit 900 searches the reception-sidedatabase for attribute data of a part corresponding to theidentification code and outputs the corresponding attribute data. Insuch a manner, by transferring the identification code of each part, theamount of the image information is largely reduced, and high-speedtransfer of the image information can be realized.

FIG. 28 shows an example of the data structure in the reception-sidedatabase 310.

For example, when reproduction of image with fidelity is intended,desirably, the contents of the reception-side database 310 are the sameas those of the database 300 on the transmission side. However, in thecase of the other purposes such as the case where only information suchas a placement state of objects is desired to be transferred promptly,the contents of the databases do not have to be always the same eachother. For example, for easy explanation for children, although it isdifferent from an object, the object may be reproduced as a partsymbolically representing the object by animation or illustration.

FIG. 29 shows parts and a reconstruction example in the case whereattribute data of parts of the same identification code in first andsecond part storage 400 and 410 are different from each other. Parts 61to 64 in FIG. 29A correspond to the same codes as the identificationcodes of the parts 51 to 54 shown in FIG. 7A, respectively.

However, as shown in FIG. 29A, the forms of the parts 61 to 64 areslightly different from those of the parts 51 to 54, respectively. Forexample, the desk 51 as a part shown in FIG. 7A is a desk having threedrawers on both sides. In contrast, the desk 61 as a part shown in FIG.29A has two right and left drawers in parallel under the top board. Theform of the chair 52 as a part and that of the chair 62 as a part arealso different from each other.

FIG. 29B shows a reconstructed image 60 of the parts 61 to 64reconstructed by adding positional information. As shown in thereconstructed image 60, the placement relations of the parts 61 to 64are the same as those of the reconstructed image 50 shown in FIG. 7.

Further, various concrete examples corresponding to the fifth embodimentwill be described in more detail with reference to FIGS. 30 to 33. FIG.30 shows, in place of the input unit 100, two systems of a video camera1 as a single viewpoint camera directed to a rotating object and a videocamera 3 as a multi-viewpoint camera or a mobile viewpoint camera whoseviewpoint is movable (time-difference multi-viewpoint camera). Both thevideo cameras 1 and 3 acquire images of an object. The video camera 1can complete image acquisition from the directions of 360° when anobject rotates once. On the other hand, the video camera 3 can obtain athree-dimensional image online. Specifically, a three-dimensional imageof an object can be obtained by acquiring two-dimensional images of thewhole circumference of the object. Images obtained by the video cameras1 and 3 are stored as image information into a two-dimensional imagefile 2 and a pseudo three-dimensional image file 4, respectively.

An image is obtained by the video camera 1 as two-dimensionalinformation from a three-dimensional information space as an object. Inthis case, the three-dimensional space is converted into two-dimensionaldata by a digital recording method. In the case where a plurality ofimages having parallax (having different viewpoints) of an object areobtained, three-dimensional information which can be recognized as apseudo three-dimensional image of which viewpoint direction is limitedcan be obtained. On the other hand, an image can be obtained by thevideo camera 3 as three-dimensional image directly from thethree-dimensional information space as an object. A moving image havingmotion parallax due to movement or a still image having no parallaxcaused by motion of the object is obtained and processed in a mannersimilar to the above and the processed image is stored in the image file2 or 4.

As shown in FIG. 30, image information of the object stored in such amanner is analyzed by an image code converting apparatus 10corresponding to the part specifying unit 500 and, after that, convertedto an information code as an ID (key) or a code in correspondence withthe kind of the information, the number of pieces of the information,and the details such as a rough position of the object, the direction ofa line segment, color, and texture so as to be associated with thedetails. Specifically, the information code converting apparatus 10 hasa correlation function computing means 11 for analyzing a field formaking an analysis on the basis of a field information database 11A, anoptimum coordinate generating means 12 for generating optimumcoordinates in an image by analyzing the result of computation of thecorrelation function computing means 11 for field analysis on the basisof a three-dimensional coordinate code database 12A, a preprocessingmeans 13 for performing an outline process or the like on an item, abody, or the like as an object whose outline in image information isclarified by analysis in the further obtained image, and a correlationfunction computing means 14 for part retrieval for making conversioninto sequence codes of items on the basis of data obtained from themeans 11, 12, and 13 and generating an information code to be combinedwith the sequence code.

In the information code converting apparatus 10 having the aboveconfiguration, the data such as outline obtained by the preprocessingmeans 13, the field data obtained by the correlation function computingmeans 11 for field analysis, the optimum coordinate data obtained by theoptimum coordinate generating means 12, and the like as bases areconverted to the sequence code of items by the correlation functioncomputing means 14 for part retrieval. In the correlation functioncomputing means 14 for part retrieval, the sequence code derived by theconversion is associated with each of information pieces regardingvarious objects to be recognized and compared and contrasted with dataregarding the object (refer to the example A of storage data in FIG. 3)preliminarily registered, generated, and stored in a three-dimensionalpart storage (database) 14A as a first part storage to select dataregarding the corresponding object, and an information code to becombined with the sequence code is generated.

In the field information database 11A in the correlation functioncomputing means 11 for field analysis, a lump of objects is classifiedas a database. For example, at the time of making frequency analysis onimage information regarding an object obtained as an image, setting canbe made so as to recognize the upper and lower sides in such a mannerthat the complicated side on which the frequency component is high isrecognized as a lower or upper side and a brighter side is recognized asan upper or lower side. The far and near sides can be also recognized insuch a manner that the side in which a frequency component is high isset as a complicated far side. It is also possible to divide a spaceinto outdoor, indoor, air, sea, and so on to thereby limit partsexisting in the divided space, store the above as parts, and divide timeinto morning, daytime, seasons, and so on.

In the case where meaning, such as placement which is impossible fromthe viewpoint of probability or contradictory placement, is generatedretrieval of the object and parts should be redone.

In the preprocessing means 13, edges are obtained to extract an outlineor the like so as to recognize the object acquired as an image, acoordinate system which facilitates arrangement of the objects isobtained, an effective three-dimensional mesh is generated, and a partis converted to edges and compared. For example, by detecting ahorizontal plane and a vertical plane, attention is paid to one part andazimuth coordinates are determined. The part whose azimuth coordinatesare determined is regarded as a part whose three-dimensional coordinatesare obtained. The azimuth coordinates can be always mutually convertedto an orthogonal coordinate system (three-dimensional stationarycoordinate system), and a conversion equation can be derived.

As a first method of forming a three-dimensional image, when a part canbe specified, the same part can be determined in a different acquiredimage. Therefore, by positioning parts (coupling of images), images canbe generated and coupled as a three-dimensional image. As a secondmethod, by moving and turning the viewpoint of the camera, the azimuthcoordinate system changes so as to follow the viewpoint of the camera.Therefore, when the azimuth coordinate system is seen from a reproducedpart sequence, the visual field of the image can be widened. On thecontrary, the amount of the moving and turning of the camera can becalculated from the movement of a part and deformation of the part, thatis, deformation of the azimuth coordinate system.

With respect to coupling of images, it is not always necessary to obtainthe camera position, movement, and rotation. As long as a part isspecified, the azimuth coordinates are unconditionally obtained fromdeformation of the specified part. That is, even when acquired imagesare different from each other, by tracing the azimuth coordinates of acertain part, images can be coupled. Therefore, the coordinate transformformula of the azimuth coordinate system in the case where the viewpointof the camera is moved can be derived.

In the correlation function computing means 14 for part retrieval, fromacquired images of the object, the analyzed and recognized object can bespecified as a part image by the correlation function computing means 11for field analysis, optimum coordinate generating means 12,preprocessing means 13, and the like. Also, the image of the object iscompared and contrasted with data regarding various parts as dataregarding the object stored in the three-dimensional part storage 14Acorresponding to the part image to select a corresponding part. Whenthere is no corresponding part, a similar part can be retrieved or theobject is measured with higher precision and registered as a new part.

In this case, in the three-dimensional storage 14A, as shown in FIG. 2,for example, when the object is recognized as “table-1,” numerical valuedata of the shape is set as “25694458,” numerical value data of thecolor is set as “2685696,” and various data of the other attributes isassociated as specific numerical value data. An information code calledID (identification code) or key is specified as, for instance, “1001.”

Similarly, for example, the other various objects recognized are storedas specific numerical values of information codes in such a manner that“1002” is stored for “table-2,” and “1003” is stored for “table-3.” Theattributes such as shape and color are similarly converted as data ofspecific numerical values. By combining the ID (key) of the recognizedobject and data regarding various attributes of the object incorrespondence with the result of analyzing and recognizing an image, aninformation code can be generated.

The data regarding the object in the three-dimensional part storage 14Abelongs to the correlation function computing means 14 for partretrieval on the transmission side and is provided as data correspondingto the information code. The correspondence between the data regardingthe object and the information code is almost similar to thecorrespondence between data and information for reproducing the objectof the three-dimensional part storage (database) 24A as a second partstorage belonging to the part sequence processing means 24 on thereception side which will be described herein later. For example, in thecase of performing conversion and reproduction on assumption that thedata regarding the object and the data reproducing the object is thesame as the information regarding the object to which information forreproducing the object is input, those data are satisfied as data havingalmost the same database configuration.

In the three-dimensional part storage 14A, as various image informationobtained by the video camera 1 or 3 as information inputting means, amatter as an expected object is modeled and stored as a part. Therefore,the attributes regarding an object, for example, physicalcharacteristics of the object such as size, characteristics of the outershape such as a corner, a curved surface, and a circle, color, material,weight, symmetry, surface reflectance, luster, smell, spectrum of sound,and life are stored in a database. Further, the other various attributessuch as numerical values indicative of danger or taste, manufacturingdate, manufacturer, and object's existing position condition such asoutdoor or indoor are also stored in a database.

The relation with another body such as affinity to another part orexclusion of another part, the relation with another body regardingcharacteristics for recognizing a body, the priority of thecharacteristics for recognizing a body, the other attributes, therelation with another body, and the like, and the other characteristicsare also arranged in order.

To an existing part, for example, to a part such as a car, various partssuch as body, tires, steering wheel, and engine are coupled. Therefore,a part is set so as to be recognized as a part even if it is constructedby the various parts.

To the three-dimensional part storage 14A, by the correlation functioncomputing means 14 for part retrieval, a learning means 15 for learninginformation regarding an object to which a code is not given in thecorrelation function computing means 14 for part retrieval can beconnected. The learning means 15 is constructed so as to register a newobject itself as a new part and register a new attribute of an object.In the case of information regarding an object which is not set in adatabase, the learning means 15 converts it to an approximateinformation code as a part having high existence probability and learnsthe converting operation.

A part sequence generating means 16 for linking an obtained image or thelike of the object each time a code is given to the image by thecorrelation function computing means 14 for part retrieval and analyzingthe coordinates of a sequence state of the object is provided. In thepart sequence generating means 16, raster/vector conversion of asequence of objects is performed so that the object is linked with theID of the part in the three-dimensional part storage 14A.

Specifically, in the information code converted by the correlationfunction computing means 14 for part retrieval, for example, in theinformation codes such as the above-described “table-1,” “table-2,” and“table-3,” the coordinates, direction, and the like on an image of eachcode are converted to IDs or keywords. That is, the direction or thelike is added to a part. Regarding formation of coordinates, theposition along each of the X, Y and Z axes is given in a numerical valuesuch as “12.236.” Similarly, the direction of each of the axes X, Y, andZ is also given in a numerical value such as “0.365.” Each of thenumerical values is stored as a sequence code into a three-dimensionalpart sequence database 16A.

For example, as shown in FIG. 9, by the coordinate database 12A in whicheach of the coordinate and direction of an object in an image analyzedand recognized such as “table-1,” “table-2,” or “table-3” is analyzedand converted to numbers, data are set as a sequence code. By making thesequence code correspond to an information code set for each object, thecoordinate, direction, and the like are combined with data such as thesequence code indicative of the contents of coordinate-X, coordinate-Y,coordinate-Z, direction-X, direction-Y, and direction-Z and the like.The resultant is stored in a coordinate code database.

In such a manner, in the information code converting apparatus 10, theobject in the image information obtained by the video camera 1 or 3 isanalyzed and recognized, compared with data regarding the object in thethree-dimensional part storage 14A, identified, and converted to aninformation code. The sequence state of parts is also arranging theimage which reproduces the parts in accordance with the sequence code(refer to FIG. 9) as keywords of the coordinates, directions, and thelike sent so as to be linked with the input image of parts, and has thepart sequence processing means 24 for arranging the image whichreproduces the arranged parts. The image which reproduces the arrangedparts is displaced as a reproduction image of the object on a display 28such as a television monitor via an image reproducing means 27 as outputmeans.

Since the part has attribute data of the three-dimensional shape, ifthere is a three-dimensional display, the image can be displayedthree-dimensionally. However, the image is generally displayed as animage of a dimension of a free viewpoint.

The part sequence processing means 24 inversely converts a specificinformation code by the second three-dimensional part storage 24A havingthe database configuration almost corresponding to the firstthree-dimensional part storage 14A. The specific information code isobtained by converting the image of the part recognized on the basis ofthe data of the object in the first three-dimensional part storage 14Aby the correlation function computing means 14 for part retrieval. Thepart sequence processing means 24 selects and converts an image whichreproduces the part corresponding to the original image from theinformation code including the attributes such as shape and color. Thatis, the part sequence processing means 24 is constructed so as toreconstruct an image of the input object obtained by the video camera 1or 3 by the vector/raster conversion of coordinates of the image whichreproduces the part together with the data of a three-dimensionalcoordinate code database 22A and linking of the image which reproducesthe part.

In this case, the sequence code as conversion data regarding partsequence is formed in a manner quite similar to that in thethree-dimensional part sequence database 16A in the part sequencegenerating means 16 (refer to FIGS. 31 and 9). In accordance with thesequence code transmitted in relation with each of the informationcodes, by a three-dimensional part sequence database 26A, thecoordinates of an image which reproduces a specific part are set in aposition along the axes X, Y, and Z. Similarly, the directions of theimage can be also arranged and reproduced as the directions of the axesX, Y, and Z.

With respect to the sequence in the case of reproducing a part, afterobtaining a stationary coordinate system, for example, the parts aresequentially adhered so as to be in contact with dominant parts such asthe ground surface, water surface, floor, wall, and desk registered asparts. At the time of the adhering operation, it is set so that rotationand movement vectors and the like of the parts are given, the direction,visual field range, movement vector, and the like as viewpointinformation of the video camera 1 at the time of acquiring the object'simage are detected, and three-dimensional coupling of images by linkingscreens can be also made possible.

At the time of arrangement for reproducing the parts, relatedinformation such as when, where, who, with whom, what, why, and how isalso set so as to be output. In this case, it is suitable so that aprocess of detecting contradiction of a part, comparing the part withanother part and selecting can be tried in consideration of the relationwith the other parts, existing conditions, and so on. Further, in thiscase, in a manner similar to the three-dimensional part storage(database) 14A in the correlation function computing means 14 for partretrieval, it is set so that learning of the image of the part and partsrelated to the image by registering, correcting, eliminating, and thelike can be performed by either a forcedly method (forced learning ofonce) or an experimentally method (statistic learning).

Even in the case where all of the information pieces regarding the inputobject are not converted to codes due to insufficiency or the like ofdata regarding the object in the three-dimensional part storage(database) 14A in the information code converting apparatus 10, when theunconverted portion is an image, it is sufficient to transmit the imageinformation as it is. Also in such a case, the transmission amount canbe reduced extremely.

In the embodiment of the information converting system according to theinvention, an input signal supplied to information input means issubjected to an imaging process and becomes a comparison signal (inputimage) which is compared with data regarding a part in a comparing anddetermining apparatus. In this case, it is desirable that the inputsignal (image signal) is processed at the highest level of the time.

However how much the input signal is processed, it does not become anoutput signal. The output signal is not information processed but isalways the part itself obtained by coincidence in the comparingoperation, a part to which a new attribute is added, or the ID codes ofthe parts. The input signal and the part have generally differentdimensions or physical quantities. For example, when the input signal isa two-dimensional image, the input signal is a two-dimensional outputsignal and a part is a three-dimensional signal.

As shown in FIG. 30, a three-dimensional signal or a pseudothree-dimensional signal may be input to the comparing and determiningapparatus (part specifying unit).

On the other hand, at the time of comparing parts, as attributes of theparts, the front, upper side, lower side, symmetry axis, center axis,and the like are determined in advance. When the front, upper side,lower side, symmetry axis, center axis, and the like can be determinedalso from the input signal, comparison is limited in an extremely narrowrange, so that it is effective. For displaying a coordinate system, theground plane, vertical plane, and horizontal plane are determined. Acomparison part candidate is clipped at the ground plane. For acomparison part, temporary front, upper side, lower side, symmetry axis,asymmetry axis, and the like are determined. At the time of actualcomparison, by making the front, upper side, lower side, and symmetryaxis closer to those, the point having the highest coincidence isdetected. The parts are not compared in an unplanned manner. The partsare approximated by an algorithm such that similar planes are madecloser to each other, a coincidence point is obtained, the degree ofcoincidence is evaluated, the parts are turned and moved in thedirection of increasing the degree of coincidence, and the finalcoincidence point is obtained. The part has, as an attribute,information regarding comparison items and the comparison priority inthe comparing and determining operation.

Next, attributes of a part and those of another part to be compared andthe relation of a set of three parts will now be described.

In this case, a group of items as an object is decomposed to a group ofparts, and each of the parts is always a block having meaning (forexample, a human being, a car, or the like). Each part can be decomposedto smaller parts (such as hand, leg, wheel, and bumper). The smallerparts are dealt as attributes of the part. Further, each block havingmeaning is decomposed to parts and, after that, determined as a setposition relation of a plurality of blocks each having meaning.

As a combination of a plurality of independent parts, a new part may beconstructed. Attributes and recognition data are given to the newlyconstructed part as a new part, and the part is registered as a newpart.

The recognition data is attribute priority data, history data,experience (place, time, seasons, circumstances, parameters, and so on),and data necessary to recognize the part and has a learning function.The recognition data is data necessary to recognize the part so as to beseparated from the other and, as a general rule, the part itself has therecognition data. The recognition data is evolved by experience andlearning. Therefore, history becomes important data for recognition.

It is not always necessary that the comparison signal is for the whole.Specifically, since the object is three-dimensional, only a part of theobject can be generally observed. In many cases, the object ispositioned behind another body and only a part of the object is seen.Therefore, a part of the object is compared with a part of the part, anddetermination is made only by coincidence of the parts. For example,since the part is three-dimensional (object), the part can be coincidedwith two-dimensional projection of only a part of the part (obtainedinformation). A three-dimensional part is compared with athree-dimensional part of the object and, by coincidence of only theparts, the part as a target can be selected.

Only a part of a two-dimensional image (or a pseudo three-dimensionalimage obtained by processing a two-dimensional image) as an input imagecan become image information for comparison (hereinbelow, also called a“comparison signal”).

Next, the attribute of a part and a coincidence determining referencewill now be described.

At the time of comparison of a part, not only the shape but also all theattributes become candidates to be compared. For example, in the case ofan image, not only the shape and coordinates of a body, color,distribution characteristic, texture, and other total information(sound, smell, hardness, and the like) are objects to be compared. Theobject is decomposed into parts, and a plurality of parts are signaledas an arrangement of the parts. By rearranging the parts, an outputsignal is constructed and reproduced. Further, the part has adetermination reference of itself as a part of the attributes. That is,each of parts has a suitable reference for determining -the coincidenceof itself, thereby remarkably improving precision.

In the information converting system according to the present invention,the kinds of functions of a comparator (part specifying unit) which isdesirable in the case of comparing data regarding the above-describedpart are as follows.

-   A. giving priority to comparison by shapes-   B. giving priority to comparison by a distribution of hue-   C. giving priority to comparison by a distribution of spatial    frequency components

Comparison and determination is made by variously combining the above A,B, and C.

In the case of giving priority to the functions, the following threekinds (1) to (3) of comparators can be considered.

-   (1) In the case of giving priority to comparison by shapes, it can    be constructed so that the shape of an input two-dimensional image    and that of a three-dimensional image in the part storage or the    shape of an input three-dimensional image and that of a    three-dimensional image in the part storage are compared with each    other, and the coincided image is output as an output signal.-   {circle around (1)} In this case, when continuous images having    parallax are used as input two-dimensional images, a pseudo    three-dimensional image can be obtained from the input images by an    imaging process. In such a manner, separation of an object is    facilitated and comparison with a part is facilitated. Also in    comparison, a three-dimensional part can be compared as it is, so    that determination can be made with extremely high precision.-   {circle around (2)} Irrespective of existence or absence of parallax    in input images, input two-dimensional images obtained from multiple    directions by a plurality of cameras are kept as they are, a    three-dimensional part is decomposed into the multiple directions    corresponding to the cameras so as to be projected    two-dimensionally, and placement of the parts so that the    multi-direction two-dimensional projection of the part and inputs in    the multiple directions (two-dimensional images in eight directions    having differences each of 45°) coincide with each other, thereby    enabling coincidence to be determined.-   {circle around (3)} Further, in the case where there is only one    two-dimensional image, it can be compared with two-dimensional    projection of a part. Specifically, in the case of simple objects    whose number of kinds is small, it is sufficient.-   {circle around (4)} By comparing the outline of an input image with    a three-dimensional outline of a part, a coincidence point of the    rough outlines can be obtained in the beginning. In this case, the    permissible deviation of the outline of each part is set to a degree    that the characteristics of the part are not lost. The outline is    given as a plurality of loci of a point of a three-dimensional    extreme. The rough outline coinciding function includes enlargement    and reduction of a part.

Subsequently, the system advances to a process of obtaining coincidenceof planes and, simultaneously, position and direction are specified. Ata stage coincidence is determined, the three-dimensional outline of apart is enlarged/reduced so as to coincide with an input image. This isa method of generating data of a new part.

-   (2) In the case of giving priority to comparison by hue    distribution, in rough shapes, distributions of hue components of    colors are preferentially compared and determined to obtain    coincidence. This is effective to natural matters such as mountains    and trees.-   (3) In the case of giving priority to comparison by distributions of    spatial frequency components, in rough shapes, distributions of    spatial frequencies are compared with each other to obtain    coincidence. In this case as well, in a manner similar to the    above-described (b), it is extremely effective to natural objects.

On the other hand, in the information converting system according to theinvention, selection of a part denotes recognition of attributes of thepart. However, in the case of only a single part or a plurality of partswhich are closely related to each other, separation of a complete partin the system of the invention does not occur.

-   (1) Therefore, decomposition of all of objects into parts denotes    that all of matters constructing the object are recognized together    with their names and attributes and, moreover, the whole is    understood. That is, since the attributes of each part are already    known, when the whole object is decomposed into parts and each    object is associated with each part, it means that the system    recognizes each of the objects. When placement of the parts is    determined, it means that the whole object as an individual object    can be understood.-   (2) If a part corresponding to an object is not found, it means that    the object cannot be recognized. The case in which an object cannot    be recognized is dealt with as follows.-   {circle around (1)} It is dealt with as a case where it is    unnecessary to recognize the object and circumstances in which the    object does not exist are determined.-   {circle around (2)} It is dealt with as a case where a part    corresponding to the object does not exist in a part storage as a    database. In this case, a part is newly generated to re-recognize    the object or the volume and occupied space of an unknown body    obtained from the input image are displayed without recognizing the    shape.

Further, by decomposing the object into parts, general characteristics(attributes) of each object and particularity of the object areimplanted as new attributes to a part, and the resultant becomes anoutput signal. That is, the general characteristics of each of the partsobtained by decomposing the object are included as attributes in thepart. Only the particularity is added as a new attribute of the part tothe output signal.

The embodiment in the information converting system according to theinvention has been described with respect to the case of obtaining animage which reproduces an object by transmitting an information codeobtained by converting an image of the object by analysis, recognition,or the like to a remote place, receiving the information code on areception side or outputting the information code as it is to anotherdevice, and inversely converting the received information code. However,information regarding the obtained object is not limited to only animage.

That is, as long as information can be converted to a general physicalquantity, by transmitting information regarding obtained various objectsas information codes, the various objects can be reproduced at thedestination of transfer. For example, information regarding each of thefive senses of visual sense, auditory sense, olfactory sense, sense oftaste, and tactile sense may be used. Further, every informationincluding properties such as quality, weight, and surface reflectance ofvarious matters, life, danger, and taste can become a target to beconverted.

FIG. 32 is an explanatory diagram showing a modeled algorithm in thecase of comparing and recognizing an input image regarding an object anda database regarding attributes of each of parts stored in the partstorage (first and second part storage 14A and 24A) in the informationconverting system according to the invention by a comparing andrecognizing apparatus (recognition engine) in the correlation functioncomputing means 14 for part retrieval shown in FIG. 30.

The explanation so far is organized and shown in FIG. 32. In FIG. 32,the process from the start of a loop of the system of the invention tothe end of the loop is as follows. First, an image and its attributesare input as image information. For the image input and attribute input,the situation and priority are determined via a search database, and acommand of selecting a situation and a classified part shelf is sent tothe first part storage 14A. On the basis of the selection command, adatabase of the attributes of parts is searched for the situation andthe classified part shelf in the part storage.

On the other hand, the image input and attribute input are subjected toa preprocess regarding ZKey (depth signal), hue, spatial frequencydistribution, or the like, and the resultant is input to the recognitionengine. In the recognition engine, according to the result of searchingthe first part storage 14A, data based on part string output andpriority determination sequentially accessed is input together with thepriority of the attributes of a part, selection of a recognizingalgorithm, and acquisition of parameters to a comparison engine(comparing unit) and compared with the input signal. A partcorresponding to a coincided part is output as a data, and all of partsconstruct a part group. The output part group constructed by the outputdata of the parts is transmitted, recorded, or displayed in accordancewith the purpose.

At this time, the second part storage 24A which stores a databaseregarding attributes of parts similar to the first part storage 14A issearched for a part which can reproduce corresponding image information,parts are rearranged, and the image information can be used, displayed.or the like on the basis of proper determination or the like.

A newly determined result with respect to the output part group can beproperly stored as record data, used as information for determiningsituation and priority, and stored as new data into a search database.

FIG. 33 is an explanatory diagram showing a modeled information processof the system configuration shown in FIGS. 30 and 31 in the informationconverting system according to the invention.

In FIG. 33, in step S1, with respect to the image information obtainedby an output of a video camera, the whole is grasped and a situation isselected. Specifically, in the step, the situation is selected for theimage input, the outline is determined by image retrieval with referenceto history from a database, and a situation organizing shelf in the partstorage is determined.

Step S2 is a separating step by ZKey indicative of the depth of thegroup of objects. To be specific, in this step, the whole image isroughly, three-dimensionally decomposed to a group of objects. The imagemay not be completely separated into corresponding parts. For example,an object may be decomposed to a lump of parts or a part of a largepart.

In step 3, an image map is generated on the basis of a hue distribution,a spatial frequency distribution, a point at infinity, texture, or thelike. In this case, a group of objects is decomposed to a plurality ofareas. It may not be complete and the areas of the same quality arepainted. An area smaller than a corresponding part can be also used.Even if areas are incomplete, they can be used extremely effectively asinformation of a clue to recognition.

In step S4, connection to a coordinate system part by vertical,horizontal, and ground planes is determined. Specifically, first,elements constructing an object image are determined with respect to thebasic configuration of a vertical line, a vertical plane, a horizontalline, a horizontal plane, a ground plane, and the like in the image.

In step S5, connection to a stationary object (described as a stationaryobject part in FIG. 33) is made by determining the upper and lower sidesand the front and detecting the symmetry axis and visual gravity. Inthis case, from the group of objects derived in step S2 and the mapgenerated in step 3, the upper and lower sides, front, symmetry axis,and visual gravity of an object are detected. With reference to them,the position and posture of the part can be narrowed down by thecomparing and recognizing apparatus.

In step S6, a velocity vector is detected for connection to a mobileobject. In this case, a mobile member is processed separately from astationary member. The velocity vector and an acceleration vector(including reciprocating motion) of the mobile member are obtained,

In step S20, the other forms of the object are obtained and comparedwith the attribute of the part or given as an attribute.

Further, in step S7, the part corresponding relation of the group ofobjects and situation is recorded. In this case, a reconstructedthree-dimensional image is recorded and used for retrieving an inputimage.

In step S8, a determination condition is added by giving a condition tothe recognition algorithm. By feeding back the recognition result,setting of a situation is simplified, and candidates of parts are alsonarrowed down. Thus, the speed of computation of comparison andrecognition is increased.

In step S9, a comparison condition is set. In this case, by comparingthe outline of an object with that of a part first, comparison byhigh-speed computation can be made. Next, planes are compared with eachother to determine the details.

In step S10, comparison and determination is made by a convergencecorrelation perturbation approaching method. Specifically, theconvergence perturbation approaching method is a method of obtaining acoincidence point by moving the visual field of a part in the directionof reducing a volume ΔV of a discrepancy portion between the visualvolume of the object group and the visual volume of the part. In otherwords, correlation is not needlessly computed for the wholethree-dimensional space which is extremely large, but information foradvancing computation of correlation in the direction of graduallyapproaching coincidence from a portion around the object is given. Insuch a manner, the speed of the computation can be extremely increased.

In step S11, an operation of the recognition engine is performed. To bespecific, the recognition engine has various recognition algorithms. Byan option given as an attribute to a part, the algorithm peculiar to thepart is selected and the part is determined. Simultaneously, aself-recognizing function of the part itself can be obtained to performa comparing and collating operation.

That is, the selection of the algorithm of comparison and recognitionand the parameters of the algorithm are received from the part. In thecase of a part having a self-recognizing function, the algorithm itselfis received from the part.

In step S12, a determination algorithm is determined (in the case of apart having no self-recognizing function).

In step S13, determination of a part, transfer of attributes, selectionof part attribute priority and a new algorithm, acquisition ofparameters from a part and, further, acquisition of the algorithm itselffrom the self-recognizing function are carried out.

In step S14, a part train is sequentially sent. In this case, from thesituation organized shelf in the part storage, parts are sequentiallysent in accordance with the priority.

In step S15, the parts are output.

In step S16, the parts are reconstructed.

Further, in step S17, a situation is determined. In this case, theposition of the situation is understood from the attributes anddistribution of a plurality of parts, and normal/abnormal, danger/safe,preferred/unpreferred, beautiful/ugly, and the like can be determined.

Finally, in step S18, an action is taken and an image is displayed. Inthis case, by receiving a result of situation determination in step S17,a proper action can be taken. Even in the case of simply displaying theresult, the proper visual angle can be automatically determined andselected.

Sixth Embodiment

With reference to FIGS. 34 to 36, a sixth embodiment of the inventionwill be described.

In the sixth embodiment, an example of using the information convertingsystem of the invention as a monitoring system for a parking lot ofcontract vehicles will be described.

A parking lot of contract cars is usually unattended. Consequently,there is a case that a vehicle other than the contract cars parkswithout permission. As measures for preventing parking withoutpermission, it is considered to keep watch on the parking lot so as toprevent parking without permission by a guard patrolling the parking lotof contract cars or by installing a monitor camera. However, to make aguard patrol the parking lot of contract cars or monitor a monitor imageall day long, personnel expenses are necessary. It is thereforedifficult to effectively prevent parking without permission at low cost.

In the sixth embodiment, an example of keeping watch on the parking lotof contract cars by the information converting system will be described.

First, referring to the functional block diagram of FIG. 34, theconfiguration of the information converting system of the sixthembodiment will be described. As shown in FIG. 34, the informationconverting system of the sixth embodiment includes an input unit 100, acomparison image generating unit 200, a part specifying unit 500, adatabase (DB) 400, an analysis information generating unit 700, and anoutput unit 600.

The input unit 100 in the sixth embodiment takes the form of a monitorcamera installed so as to see the whole parking lot of contract cars.

In the database 300, car models and shapes of various vehicles arestored as attribute data. Further, in the database 300, as attributedata of each block in the parking lot of contract cars, the car modeland shape of the contract vehicle are registered.

The comparison image generating unit 200 generates image information forcomparison including the images of vehicles parked in images of themonitor camera.

The part specifying unit 500 specifies a part corresponding to theparked vehicle in the image of the monitor camera, thereby recognizingthe car model of the parked vehicle.

The analysis information generating unit 700 generates analysisinformation of a group of parts which are different from each other bycombining data of specific items in the attribute data of the parts. Inthe sixth embodiment, the analysis information generating unit 700compares and collates data of a specific item which is the car modelamong the attribute data of the parked vehicle with data of a specificitem which is the car model of the contracted vehicle in the attributedata of the parking block. The analysis information generating unit 700generates analysis information indicating whether the parked vehicle isthe contract vehicle or a vehicle parked without permission.

The output unit 600 makes a report or outputs warning when the analysisinformation indicates parking without permission.

Referring to the flowchart of FIG. 35, the processes in the case ofapplying the invention to the monitoring system for the parking lot ofcontract cars will be described.

First, by the monitor camera as the input unit 100, an image of thewhole parking lot of contract cars is obtained as shown in FIG. 36A(step S1 in FIG. 35).

FIG. 36A shows an image of first to third blocks 71 to 73 of the parkinglot and a mini-truck 70 parked in the second block 72. Since the vehicle70 is included in the image (“Yes” in step S2 in FIG. 35), the partspecifying unit 500 recognizes the car model of the parked vehicle onthe basis of the attribute data in the database 300 (step S3 in FIG.35).

Subsequently, the parking position of the vehicle 70 is extracted (stepS4 in FIG. 35). In this case, as the parking position, the secondparking block 72 is extracted.

Next, the part specifying unit 500 retrieves and reads the attributedata of a contract vehicle 75 in the second parking block 72 from thedatabase 300 (step S5 in FIG. 35). In this case, as the item of “carmodel” in the attribute data, a “wagon” 75 as a part shown in FIG. 36Bis read.

By the analysis information generating unit 700, the car model of thevehicle 70 in the image is compared and contrasted with that of thecontract vehicle 75 registered (step S6 in FIG. 35).

At the time of comparison and contrast, the shapes and colors of thewhole vehicles may be directly compared with each other, or the carmodel may be specified from the shapes of parts of the vehicle and theplacement relation of the parts.

For example, (1) each of the parts constructing the car such as wheels,headlight, fender, doors, windows, tail lamps, and number plate arespecified first.

(2) Subsequently, the placement relation of the parts constructing thecar is reconstructed.

(3) When the three-dimensional positional relation of the parts matchesthe condition, the object is recognized as a car. Preferably, the carmodel of the car is further specified.

Such a method is particularly suitable for recognizing an object whoseshape is not fixed or an object whose shape is easily deformed.

In the embodiment, although the parked vehicle is the mini-truck 70, thecontract vehicle is the wagon 75, so that the car models do not coincidewith each other (“No” in step S7 in FIG. 35).

In this case, the analysis information generating unit 700 determinesthat the vehicle 70 parked in the second parking block 72 parks withoutpermission (step S9 in FIG. 35).

In other words, the analysis information generating unit 700 newlygenerates analysis information that “the parked vehicle parks withoutpermission” on the basis of the attribute data of the car model“mini-truck” of the parked vehicle 70 specified from the image and theattribute data of the car model “wagon” of the contract vehicle 75 ofthe second parking block 72.

Subsequently, the output unit 600 outputs a message that there is thevehicle parking without permission by, for example, reporting it to themanager of the parking lot of contract cars or announcing a warning thatparking without permission is prohibited.

When the car model of the parked vehicle coincides with that of thecontract vehicle registered, the analysis information generating unit700 generates analysis information that “the parked vehicle is thecontract vehicle” (step S8 in FIG. 35).

As described above, according to the sixth embodiment, only byinstalling one monitor camera in the parking lot, parking withoutpermission in the whole parking lot can be automatically watched. Suchan automatic monitoring system cannot be realized without recognizingthe car model of a parked vehicle.

A method of verifying the number plate of a vehicle can be alsoconsidered. However, particularly when a parking lot is large, it isdifficult to read the number plates of all vehicles parked by one or asmall number of monitor cameras.

When not only the car model but also elements such as color of a vehicleare added as elements for determination, the determination precision canbe further increased.

The ground of a parking lot is generally a known flat surface or a knowngentle slope. The tires of a parked vehicle are in contact with the flatsurface. Therefore, as an intersection between the direction of avehicle seen from the monitor camera and a known flat surface, theposition of the vehicle can be obtained.

Seventh Embodiment

A seventh embodiment of the invention will now be described withreference to FIGS. 37 to 39.

In the seventh embodiment, an example of using the informationconverting system of the invention as a register system in a store willbe described.

In a store such as a convenience store, for management and checkout ofcommodities, a bar code is attached to each commodity. A bar code systemof identifying each commodity by reading the bar code at a checkout andautomatically displaying the total amount of commodities is inwidespread use.

In the bar code system, bar codes are not preliminarily attached to allof the commodities or packages of the commodities. Consequently, in manycases, a bar code has to be attached to each of the commodities in astore. As a result, particularly in a large store such as a supermarket,there is a case such that enormous efforts are required to attach barcodes to a large number of commodities.

At a checkout, conventionally, a clerk picks up commodities one by oneto pass the commodity by a fixed bar code reader so that the bar code isread. Because of this operation, a burden is therefore applied to thearms and lower back of the clerk. A handy-type bar code reader hastherefore been developed and a way of allowing a bar code to be read byputting the bar code reader to a commodity is in widespread use. Also inthis way, however, a clerk still has to put the bar code reader to thecommodities one by one so that the bar code is read.

Moreover, in the bar code system, the object actually identified is abar code attached to a commodity. That is, the commodity itself is notdirectly identified. Because of this, it is feared that, when anerroneous bar code is attached to a commodity, the commodity iserroneously identified at a checkout, and a wrong amount is displayed.

In the seventh embodiment, therefore, the information converting systemis applied as a register system.

First, by referring to the functional block diagram of FIG. 37, theconfiguration of the information converting system of the seventhembodiment will be described. As shown in FIG. 37, the informationconverting system of the seventh embodiment includes an input unit 100,a comparison image generating unit 200, a part specifying unit 500, adatabase (DB) 400, an analysis information generating unit 700, and anoutput unit 600.

The input unit 100 in the seventh embodiment is constructed by a camera110 and a scale 120. The camera 110 can take an image of the wholebasket C in which commodities are put and which is placed at a checkout.The scale 120 can measure the weight of the basket C in which thecommodities are put.

It is desirable to install a plurality of cameras 110 to acquire imagesof the basket C from a plurality of different directions.

In the database 300, as attribute data of each of commodities such as amilk package and a shampoo sold in the store, data such as the shape,price, and weight of each of the commodities is stored.

The part specifying unit 500 recognizes each of the commodities from theimages of the commodities in the basket C obtained by the camera 100.

The analysis information generating unit 700 is constructed by atotaling unit 710 and a determining unit 720. The totaling unit 710 addsup the prices of the commodities as parts recognized by the partspecifying unit 500 and generates analysis information which is thetotal amount. Further, the totaling unit 710 adds the weight of thecommodities as parts and the weight of the basket C to thereby alsogenerate analysis information of total weight.

The determining unit 720 compares and collates the weight of the wholebasket C including the commodities measured by the scale 120 with thetotal weight calculated by the totaling unit 710 and determines whetheror not the weights coincide with each other within a permissible range.When the weights coincide with each other, the determining unit 720determines that all of the commodities are correctly recognized. On theother hand, when the weights do not coincide with each other, thedetermining unit 720 determines that there is a commodity which is notcorrectly recognized in the basket C. In such a manner, the determiningunit 720 also generates analysis information of determination from themeasured weights and the total weight.

When the determining unit 720 determines that the commodities arecorrectly recognized, the output unit 600 outputs the total amountcalculated by the totaling unit 710. On the other hand, when thedetermining unit 720 determines that the commodities are not correctlyrecognized, the output unit 600 outputs an error message.

Referring to the flowchart of FIG. 38, processes performed in the caseof applying the present invention to a register system will now bedescribed.

First, by the monitor camera 110 in the input unit 100, as shown in FIG.39A, an image of the whole basket C in which commodities are put isobtained (step S1 in FIG. 38). FIG. 39A schematically shows a state Inwhich a liquid detergent 81 in a container, a Tofu package 82, and amilk package 83 are put in the basket C.

Simultaneously with the acquisition of the image, weight G1 of the wholebasket C is measured by the scale 120 (step S2 in FIG. 38).

Subsequently, by the part specifying unit 500, parts corresponding tothe objects in the image obtained by the camera 110 are specified (stepS3 in FIG. 38).

FIG. 39B schematically shows specified parts corresponding to thecommodities. In this case, liquid detergent 81 a as a part, a Tofupackage 82 a as a part, and a milk package 83 a as a part are specified.

After confirming that all of the objects in the image are extracted(step S4 in FIG. 38), prices of the specified parts 81 a to 83 a aretotaled by the totaling unit 710 to calculate the total amount (step S5in FIG. 38).

Subsequently, by adding up the weights of the parts 81 a to 83 a and thebasket C, a total weight G2 is calculated (step S6 in FIG. 38).

The measured weight G1 measured by the scale 120 is compared andcollated with the total weight G2 calculated by the totaling unit 710 bythe determining unit 720 (step S7 in FIG. 38).

When the measured weight G1 and the total weight G2 coincide with eachother (in the case of “Yes” in step S8 in FIG. 38), the output unit 600displays the total amount (step S9 in FIG. 38).

On the other hand, when the measured weight G1 and the total weight G2do not coincide with each other (in the case of “No” in step 8), theoutput unit 600 displays an error message (step S10 in FIG. 38).

As described above, by applying the invention to the register system, apart corresponding to each commodity can be specified, so that bar codesbecome unnecessary. Consequently, a clerk does not have to make barcodes attached to commodities read one by one, and the total amount ofthe commodities can be calculated in short time. As a result, theprocess at the checkout becomes quicker, and waiting time at thecheckout is shortened. Since commodities do not have to be picked up oneby one at the time of checkout, the physical burden on clerks can belessened.

Eighth Embodiment

With reference to FIGS. 40 to 42, an eighth embodiment of the inventionwill now be described.

In the eighth embodiment, an example of using the information convertingsystem for a survey on traffic volume will be described.

Conventionally, to collect traffic volume data, by employing temporaryworkers or the like, traveling vehicles and the like are counted one byone by a counter on a kind-by-kind basis.

However, when human beings do the counting, a counting error oftenoccurs. Moreover, it is difficult to verify the authenticity of totaleddata. Particularly, at an intersection of heavy traffic, because anumber of vehicles simultaneously start driving as soon as the signal isswitched, an oversight in counting often occurs.

Also by employing temporary workers, personnel expenses occur, and thesurvey cost becomes high. Especially, for a survey on traffic volume atnight or early in the morning, in many cases, it is difficult to assurethe number of workers.

In the eighth embodiment, therefore, an example of applying theinformation converting system to a survey on traffic volume will bedescribed.

First, referring to the functional block diagram of FIG. 40, theconfiguration of the information converting system of the eighthembodiment will be described. As shown in FIG. 40, the informationconverting system of the eighth embodiment includes an input unit 100, acomparison image generating unit 200, a part specifying unit 500, adatabase (DB) 300, a comparison part generating unit 400, an analysisinformation generating unit 700, an item selecting unit 800, and anoutput unit 600.

The input unit 100 of the eighth embodiment takes the form of a monitorcamera for capturing images of vehicles passing on a bridge 90schematically shown in FIG. 42. The visual field of the monitor cameramay be set to be either wide so that the whole bridge 90 is captured ornarrow so that only one traveling vehicle is captured. The monitorcamera may be installed to capture an image of a vehicle passing thebridge 90 from the front or from a side.

In the database 300, attribute data of the car models and forms ofvarious vehicles is stored.

The part specifying unit 500 specifies a part corresponding to an imageof a vehicle obtained by the monitor camera as the input unit 100.

The item selecting unit 800 selects parts whose attribute data arecombined and/or specific items of the parts in accordance with theanalysis information generated by the analysis information generatingunit.

The analysis information generating unit 700 generates analysisinformation in accordance with a result of selection of the itemselecting unit 800. In the embodiment, the models of the vehicles asparts are classified, and the number of vehicles passed is totaled modelby model.

The output unit 600 outputs the number of vehicles passed on themodel-by-model basis.

Referring now to the flowchart of FIG. 41, processes performed in thecase of applying the invention to a survey on traffic volume will bedescribed.

First, by the monitor camera as the input unit 100, an image of vehiclespassing on the bridge is input (step SI in FIG. 41).

Subsequently, by the part specifying unit 500, parts corresponding toeach of the vehicles are specified. Based on the attribute data of theparts specified, the car model is determined (step S2 in FIG. 41).

It is sufficient to set the classification of car models in accordancewith the purpose of a survey on traffic volume. For example, the carmodels can be simply classified into “heavy duty vehicles” and “nonheavy-duty vehicles” or classified by manufacturer of a vehicle.

Subsequently, in the analysis information generating unit 700, the carmodels are classified (step S3 in FIG. 41) and the number of vehiclespassed is totaled by car model (step S4 in FIG. 41).

The total value of the number of vehicles passed model by model isoutput from the output unit 600 (step S5 in FIG. 41).

In such a manner, according to the eighth embodiment, the number ofvehicles passed can be automatically totaled model by model. Thus, veryreliable data can be collected at low cost. The totaling whileperforming such classification cannot be realized without specifying thecar model of a passing vehicle.

Ninth Embodiment

Referring now to FIGS. 40, 42, and 43, a ninth embodiment of theinvention will be described.

In the ninth embodiment, an example of managing the safety of a bridgeby using the same system configuration as that of the foregoing eighthembodiment will be described.

As shown in the flowchart of FIG. 43, also in the ninth embodiment, in amanner similar to the foregoing eighth embodiment, an image of vehiclespassing on the bridge is input (step S1 in FIG. 43). However, in theninth embodiment, images of not only the vehicles on the bridge but alsoa vehicle approaching the bridge are captured.

Subsequently, in a manner similar to the eighth embodiment, models ofvehicles 91, 92, and 93 in the image are recognized (step S2 in FIG.43).

In the ninth embodiment, the item selecting unit 800 instructsgeneration of analysis information for controlling the safety of thebridge. For this purpose, the analysis information generating unit 700calculates total weight G (=g1+G2+g3) of weights g1 and g2 of thevehicles 91 and 92 passing on the bridge 90 and weight g3 of the vehicle90 approaching the bridge 90 (step S3 in FIG. 43).

That is, the total weight G is analysis information of the vehicles 91,92, and 93.

As the weight of each of vehicles, it is desirable to set a value of aload carried on the vehicle is added. For example, in the case of atruck, it is preferable to add the maximum payload.

Further, by the analysis information generating unit 700, the totalweight G is compared with a withstand load Gh of the bridge 90 (step S5in FIG. 43).

When the total weight G is equal to or less than the withstand load Gh,the bridge is determined to be safe (step S6 in FIG. 43).

In other words, the analysis information generating unit 700 furthergenerates, as determination for safety, new analysis information of thetotal weight of the vehicles 91, 92, and 93 and the bridge 90 bycombining the total weight G as analysis information and the withstandload Gh as attribute data of the bridge 90.

On the other hand, when the total weight G exceeds the withstand loadGh, the bridge is determined to be dangerous (step S7 in FIG. 43).

In other words, the analysis information generating unit 700 furthergenerates, as determination of danger, new analysis information of thetotal weight of the vehicles 91, 92, and 93 and the bridge 90 bycombining the total weight G as analysis information and the withstandload Gh as attribute data of the bridge 90.

When the bridge is determined to be dangerous, a control signal forswitching display of a signal 94 to the red signal light is immediatelyoutput from the output unit 600.

In such a manner, the safety control of the bridge can be automaticallyperformed.

In the ninth embodiment, the vehicle weight is information which is notdirectly obtained from a monitor image. Consequently, even when a humanbeing simply watches the monitor camera, safety or danger cannot bedetermined unlike the embodiment. In contrast, in the embodiment, avehicle in an image is recognized as a vehicle as a part havingattribute data, so that the safety control can be automaticallyperformed by using the weight data of each vehicle, which is notincluded in the input image. Therefore, in the ninth embodiment, moreadvanced safety control as compared with the case where the bridge issimply watched by a human being can be realized.

Although the example of applying the invention under specific conditionshas been described in each of the foregoing embodiments, the inventioncan be variously modified. For instance, in the foregoing embodiment,the example of using a video image of an object as input information hasbeen described. However, in the invention, the input information is notlimited to an image. For example, information obtained by adding soundinformation of an object or information such as temperature of theobject to an image may be used as input information.

As input information, a result of measurement of various physicalquantities such as mass, electricity, and magnetism of the object mayalso be used. An operation pattern or the like of the object may also beused.

The input image is not limited to an image of visible rays. For example,as an input image, an infrared image, an exposure image formed byradiation such as X-rays or neutron rays, or an electron-ray image maybe used. For instance, a reflected-wave image of ultrasonic wave, radiowave or the like or further a diffracted image may be used.

In the invention, by registering a pronounced sentence or word as a partand obtaining sound information in place of the image information, soundrecognition can be also performed. In the sound recognition, input soundinformation is an electric signal of sound captured by a microphone.Consequently, as sound information for comparison, a frequency spectrumas a function of time, a sound level (intensity) and, further, FM and AMcomponents obtained by analysis may be generated in consideration ofcharacteristics of a language. On the other hand, as sound partinformation for comparison, in the case of a language, frequency spectraof a word and a sentence, sound level, FM components, and AM componentsmay be given. By the part specifying unit, the correspondence betweenthe sound information for comparison and the sound part information forcomparison is checked.

In each of the foregoing embodiments, the example of performinginformation conversion in order to realize one or a plurality ofspecific purposes has been described. However, the informationconverting system of the invention can also be used as a general systemwhose use is not specified. For example, the function of an electronicencyclopedia for receiving information of an image of an object andoutputting various attribute data of parts corresponding to the objectcan also be realized by a configuration of a database, an input unit,and a part specifying unit.

INDUSTRIAL APPLICABILITY

The invention is suitable for use in various fields in which an imagerecognizing technique of a monitoring system such as a security systemcan be used. The invention is also suitable for use as an imageprocessing technique in factory automation (FA). Further, the use of theinvention as the basic technique of the information processing techniquesuch as artificial intelligence into various fields can be expected.

The invention is also suitable for use in various fields of imageinformation transmitted such as relay broadcasting.

1. An information converting system comprising: a database in whichattribute data including three-dimensional shape data and anidentification code of each of parts modeled on various objects areregistered; a comparison part generating unit for generating one pieceor plural pieces of part information for comparison from said attributedata for each of said parts; an input unit for obtaining an input imageincluding an object image; a comparison image generating unit forgenerating image information for comparison in which information piecesof said objects are not individually clipped, by performing an imagingprocess on said input image; a part specifying unit for specifying apart corresponding to said object image by using the part informationfor comparison and the image information for comparison each having thesame kind of data; and an output unit for outputting the identificationcode and at least a part of the attribute data of the specified part asa result of recognition of said object image, and said part specifyingunit comprising: a retrieving unit for retrieving a correspondingportion, which corresponds to at least a part of said part informationfor comparison, from said image information for comparison sequentiallywith respect to one or plural parts of part information for comparisonof one or plural parts; a recognizing unit for recognizing saidcorresponding portion in said image information for comparison as anobject image; and a specifying unit for specifying a part having saidpart information for comparison as a part corresponding to said objectimage.
 2. The information converting system according to claim 1,wherein said comparison part generating unit decomposes as said partinformation for comparison, the attribute data of said part into basicelements of an outline to generate basic elements or a composite elementobtained by combining a plurality of basic elements, said comparisonimage generating unit extracts the basic elements of an outline andgenerates a set of basic elements or composite elements as said imageinformation for comparison, and said retrieving unit retrieves a partcorresponding to the basic element or composite element of said partfrom said image information for comparison.
 3. The informationconverting system according to claim 2, wherein said comparison partgenerating unit generates basic elements or a composite element of acharacteristic portion of the attribute data of a part as said partinformation for comparison, said retrieving unit retrieves a partcorresponding to the basic element or composite element of saidcharacteristic portion from said image information for comparison, andsaid recognizing unit detects, after the portion corresponding to thebasic element or composite element of said characteristic portion isretrieved, correspondence between said corresponding portion and a basicelement or composite element out of the characteristic portion in thesame part, and recognizes the corresponding portion as an object image.4. The information converting system according to claim 2, wherein saidcomparison part generating unit generates, as said part information forcomparison, an element extracting filter taking the form of atwo-dimensional matrix or a three-dimensional matrix in which a highpoint is given to a pixel coinciding with the shape of said basicelement or composite element and a low point is given to a pixel apartfrom the shape of said element, and said retrieving unit retrieves, assaid corresponding portion, a portion in which the total point of pixelscoinciding with the basic element or composite element in said imageinformation for comparison is the highest.
 5. The information convertingsystem according to claim 2, wherein said comparison part generatingunit gives information for specifying only a coupling relation of saidbasic elements to said composite element, and said part specifying unitretrieves said corresponding portion on a condition that at least a partof said coupling relation coincides with said corresponding portion. 6.The information converting system according to claim 1, wherein theattribute data of each part registered in said database includesself-specifying information for instructing a method of specifying thepart, said comparison part generating unit generates part informationfor comparison for designating said self-specifying information andoutputs said part information for comparison to said part specifyingunit in accordance with priority designated by said self-specifyinginformation, and said part specifying unit specifies a part on the basisof said self specifying information.
 7. The information convertingsystem according to claim 1, wherein as attribute data of a set ofparts, identification codes and a combination condition of a pluralityof parts constructing the part set are registered in said database, andwhen specified parts satisfy said combination condition, said specifyingunit further specifies a part set obtained by combining specified parts.8. The information converting system according to claim 1, wherein saiddatabase has, as attribute data of a four-dimensional part modeled on aseries of operations of an object, a set of three-dimensional shape datain a time-series order of the object.
 9. The information convertingsystem according to claim 1, wherein said database has, as attributedata of general parts modeled commonly on an object group, attributedata common to parts modeled on the objects of the object group.
 10. Theinformation converting system according to claim 9, wherein said generalparts and parts commonly having the attribute data of the general partsare associated with each other in said database, said comparison partgenerating unit generates part information for comparison with respectto said general parts, and when said general part is specified by saidspecifying unit, said comparison part generating unit generates partinformation with respect to a part associated with the general part. 11.The information converting system according to claim 1, wherein saiddatabase captures data obtained from a recognized object image asattribute data of a specified part or replaces the data obtained from arecognized object image with a part of attribute data.
 12. Theinformation converting system according to claim 1, wherein a pluralityof parts are grouped for each set situation in said database, and whensaid input image corresponds to any of set situations, said comparisonpart generating unit generates said part information for comparison fora part in the group of the corresponding set situation.
 13. Theinformation converting system according to claim 1, wherein saidretrieving unit limits a retrieval range in said image information forcomparison in accordance with a scene of an input image.
 14. Theinformation converting system according to claim 1, wherein a pluralityof said input units obtain input images of the same object from knowndirections which are different from each other, said comparison imagegenerating unit generates image information for comparison includingtwo-dimensional shape data from each of the input images obtained by theinput units, said comparison part generating unit generates partinformation for comparison having two-dimensional shape data obtained byprojecting three-dimensional shape data of a part into the knowndirections, and said part specifying unit specifies a part of each imageinformation for comparison and confirms that the same part is specifiedabout each of the image information for comparison.
 15. The informationconverting system according to claim 1, wherein said input unit obtainsan input image including an object image photographed from a singledirection, said comparison image generating unit generates imageinformation for comparison including two-dimensional shape data fromsaid input image, and said comparison part generating unit generatespart information for comparison having two-dimensional shape dataobtained by projecting the three-dimensional shape data of said partinto an arbitrary direction.
 16. The information converting systemaccording to claim 1, wherein said input unit obtains input imageshaving parallax of the same object photographed from directions whichare different from each other, said comparison image generating unitgenerates image information for comparison including three-dimensionalshape data from each of the input images, and said comparison partgenerating unit generates part information for comparison havingthree-dimensional shape data of a part.
 17. The information convertingsystem according to claim 1, wherein said part specifying unit has asettling unit for determining a three-dimensional shape of a specifiedpart and three-dimensional coordinates indicative of an arrangementrelation.
 18. The information converting system according to claim 17,wherein when the same part is specified with respect to a plurality ofdifferent object images by said part specifying unit, said settling unitadds identifiers which are different from each other to identificationcodes of the specified parts.
 19. The information converting systemaccording to claim 17, wherein when said input image is a moving imageconstructed by a plurality of frames, said part specifying unitspecifies a part with respect to one of the frames and repeatedlyperforms only said settling process with respect to the part oncespecified for the other frames.
 20. The information converting systemaccording to claim 17, wherein said output unit reconstructs a pluralityof parts subjected to the settling process in said part specifying unitand three-dimensional space arrangement of the parts as an image seenfrom a viewpoint in an arbitrary position and displays the resultant.21. The information converting system according to claim 1, wherein saidinput unit obtains an overlapped portion of three-dimensional spaces inimage capturing ranges of input images on the basis of an object imagein each of input images of an object whose three-dimensional shape andposition are known, photographed from directions which are differentfrom each other, aligns the overlapped portions so as to coincide witheach other on a three-dimensional coordinate system, thereby couplingthe images, and obtains a viewpoint position and an angle of view ofeach of the input units.
 22. The information converting system accordingto claim 1, further comprising: a transmitting unit for transmitting anidentification code output from said output unit to a communicationline; a receiving unit for receiving said identification code; areception-side database in which said identification code and attributedata are associated with each other and registered; and a reconstructingunit for searching attribute data of a part corresponding to saididentification code from said reception-side database and outputting theattribute data corresponding to said identification code.
 23. Theinformation converting system according to claim 22, whereinthree-dimensional shape data of parts of the same identification code insaid database on a transmission side and in said reception-side databaseare different from each other.
 24. The information converting systemaccording to claim 1, further comprising an analysis informationgenerating unit for combining attribute data of a plurality of partsspecified by said part specifying unit to thereby generate analysisinformation regarding a group of the parts.